FORCE11_Workshop_Introduction_4-17-2016.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# FORCE 11\n",
    "## Exploring the data in SHARE\n",
    "\n",
    "Introduction to SHARE Queries and the current state of SHARE Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Names are difficult to disambiguate\n",
    "\n",
    "SHARE attempts to break names into Given, Family, and Additional pieces. The [SHARE person schema](https://github.com/CenterForOpenScience/SHARE-Schema/blob/master/share.yaml#L42) also includes spots for ```email```, ```affiliation```, and any links to other identifiers, such as ORCIDS, in the ```sameAs``` field.\n",
    "\n",
    "Let's do a query to showcase different names appearing in the 5 most recent documents in SHARE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import furl\n",
    "import requests\n",
    "\n",
    "\n",
    "def query_share(size, query=None):\n",
    "    SHARE_API = 'https://osf.io/api/v1/share/search/'\n",
    "    search_url = furl.furl(SHARE_API)\n",
    "    search_url.args['size'] = size\n",
    "    search_url.args['sort'] = 'providerUpdatedDateTime'\n",
    "    if query:\n",
    "        search_url.args['q'] = query\n",
    "    return requests.get(search_url.url).json()\n",
    "\n",
    "def print_title_contributors(results):\n",
    "    for result in results['results']:\n",
    "        print(result['title'].encode('utf-8'))\n",
    "        print('~~~~~~~~~')\n",
    "        for name in result['contributors']:\n",
    "            print(name['name'])\n",
    "        print('-------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistical Interpretation including the APPROPRIATE Statistical Tests\n",
      "~~~~~~~~~\n",
      "Vsevolozhskaya, Olga A.\n",
      "-------------------------------------------\n",
      "Estimated Probability of Becoming Alcohol Dependent: Extending a Multiparametric Approach\n",
      "~~~~~~~~~\n",
      "Vsevolozhskaya, Olga A.\n",
      "Anthony, James C.\n",
      "-------------------------------------------\n",
      "Rethinking Technical Services: New Frameworks, New Skill Sets, New Tool, New Roles\n",
      "~~~~~~~~~\n",
      "Eden, Brad\n",
      "-------------------------------------------\n",
      "Partnerships and New Roles in 21st-Century Academic Libraries: Collaborating, Embedding, and Cross-Training for the Future\n",
      "~~~~~~~~~\n",
      "Eden, Brad\n",
      "-------------------------------------------\n",
      "Creating Research Infrastructures in 21st-Century Academic Libraries: Conceiving, Funding, and Building New Facilities and Staff\n",
      "~~~~~~~~~\n",
      "Eden, Brad\n",
      "-------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "results =  query_share(5)\n",
    "print_title_contributors(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Names don't always show up in the same format\n",
    "\n",
    "Let's choose a name, remove the middle initial, and see if we get a result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sharepa import ShareSearch\n",
    "\n",
    "def search_a_name(name):\n",
    "    name_search = ShareSearch()\n",
    "    name_search = name_search.query(\n",
    "        {\n",
    "            \"bool\": {\n",
    "                \"should\": [\n",
    "                    {\n",
    "                        \"match\": {\n",
    "                            \"contributors.name\": {\n",
    "                                \"query\": name, \n",
    "                                \"operator\": \"and\",\n",
    "                                \"type\" : \"phrase\"\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        }\n",
    "    )\n",
    "    \n",
    "    return name_search\n",
    "\n",
    "def print_name_results(name):\n",
    "    search = search_a_name(name)\n",
    "    if search.count() == 1:\n",
    "        print('There is {} document with the contributor {}'.format(  search.count(), name))\n",
    "    else:\n",
    "        print('There are {} documents with the contributor {}'.format(search.count(), name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 8 documents with the contributor Martone, Maryann E.\n",
      "There are 10 documents with the contributor Martone, Maryann\n",
      "There are 2 documents with the contributor Maryann Martone\n",
      "There are 26 documents with the contributor Maryann E Martone\n"
     ]
    }
   ],
   "source": [
    "print_name_results('Martone, Maryann E.')\n",
    "print_name_results('Martone, Maryann')\n",
    "print_name_results('Maryann Martone')\n",
    "print_name_results('Maryann E Martone')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Identifiers are Difficult to Identify\n",
    "\n",
    "Here's a query that will search for all documents with contributors that have at least one orcid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 29413 results\n",
      "----------\n",
      "An arithmetic Zariski pair of line arrangements with non-isomorphic fundamental group\n",
      "crossref\n",
      "~~~~~~~~~\n",
      "Enrique Artal Bartolo - [u'http://orcid.org/0000-0002-8276-5116']\n",
      "José Ignacio Cogolludo-Agustín - [u'http://orcid.org/0000-0003-1820-6755']\n",
      "Benoît Guerville-Ballé - []\n",
      "Miguel Marco-Buzunáriz - [u'http://orcid.org/0000-0002-6750-8971']\n",
      "-------------------------------------------\n",
      "Discovering biomarkers for antidepressant response: protocol from the Canadian biomarker integration network in depression (CAN-BIND) and clinical characteristics of the first patient cohort\n",
      "crossref\n",
      "~~~~~~~~~\n",
      "Raymond W. Lam - []\n",
      "Roumen Milev - []\n",
      "Susan Rotzinger - []\n",
      "Ana C. Andreazza - []\n",
      "Pierre Blier - []\n",
      "Colleen Brenner - []\n",
      "Zafiris J. Daskalakis - []\n",
      "Moyez Dharsee - []\n",
      "Jonathan Downar - []\n",
      "Kenneth R. Evans - []\n",
      "Faranak Farzan - []\n",
      "Jane A. Foster - []\n",
      "Benicio N. Frey - []\n",
      "Joseph Geraci - []\n",
      "Peter Giacobbe - []\n",
      "Harriet E. Feilotter - []\n",
      "Geoffrey B. Hall - []\n",
      "Kate L. Harkness - []\n",
      "Stefanie Hassel - []\n",
      "Zahinoor Ismail - []\n",
      "Francesco Leri - []\n",
      "Mario Liotti - []\n",
      "Glenda M. MacQueen - []\n",
      "Mary Pat McAndrews - []\n",
      "Luciano Minuzzi - []\n",
      "Daniel J. Müller - []\n",
      "Sagar V. Parikh - []\n",
      "Franca M. Placenza - []\n",
      "Lena C. Quilty - []\n",
      "Arun V. Ravindran - []\n",
      "Tim V. Salomons - []\n",
      "Claudio N. Soares - []\n",
      "Stephen C. Strother - []\n",
      "Gustavo Turecki - []\n",
      "Anthony L. Vaccarino - []\n",
      "Fidel Vila-Rodriguez - []\n",
      "Sidney H. Kennedy - [u'http://orcid.org/0000-0001-5339-7185']\n",
      "-------------------------------------------\n",
      "Isolation of mouse internal organs for molecular and histopathological studies\n",
      "crossref\n",
      "~~~~~~~~~\n",
      "Vivek K. Bajpai - [u'http://orcid.org/0000-0003-0423-6195']\n",
      "Irfan A. Rather - [u'http://orcid.org/0000-0003-2752-0519']\n",
      "Kangmin Kim - [u'http://orcid.org/0000-0001-5052-5938']\n",
      "-------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "recent_results = query_share(3, 'contributors.sameAs:*orcid*')\n",
    "\n",
    "print('There are {} results'.format(recent_results['count']))\n",
    "print('----------')\n",
    "for result in recent_results['results']:\n",
    "    print(result['title'].encode('utf-8'))\n",
    "    print(result['shareProperties']['source'])\n",
    "    print('~~~~~~~~~')\n",
    "    for name in result['contributors']:\n",
    "        print('{} - {}'.format(name['name'].encode('utf-8'), name['sameAs']))\n",
    "    print('-------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Try the same for another form of identifier - an email address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 52582 results\n",
      "----------\n",
      "Data from: Contemporary and historic factors influence differently genetic differentiation and diversity in a tropical palm\n",
      "dataone\n",
      "~~~~~~~~~\n",
      "Galetti, Mauro - None\n",
      "Ribeiro, Milton C. - None\n",
      "Carvalho, Carolina C. - carolina.carvalho@ymail.com\n",
      "Collevatti, Rosane G. - None\n",
      "Côrtes, Marina C. - None\n",
      "-------------------------------------------\n",
      "Data from: Asthma-like symptoms in homeless children in the Greater Paris area in 2013: prevalence, associated factors and utilization of healthcare services in the ENFAMS survey\n",
      "dataone\n",
      "~~~~~~~~~\n",
      "Marguet, Christophe - None\n",
      "Lefeuvre, Delphine - lefeuvredel@gmail.com\n",
      "Delmas, Marie-Christine - None\n",
      "Vandentorren, Stéphanie - None\n",
      "Chauvin, Pierre - None\n",
      "-------------------------------------------\n",
      "Data from: Trans-species variation in Dmrt1 is associated with sex determination in four European tree-frog species\n",
      "dataone\n",
      "~~~~~~~~~\n",
      "Brelsford, Alan - alan.brelsford@unil.ch\n",
      "Perrin, Nicolas - None\n",
      "Dufresnes, Christophe - None\n",
      "-------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Contributors with Email Addresses\n",
    "\n",
    "results = query_share(3, 'contributors.email:*')\n",
    "\n",
    "print('There are {} results'.format(results['count']))\n",
    "print('----------')\n",
    "for result in results['results']:\n",
    "    print(result['title'].encode('utf-8'))\n",
    "    print(result['shareProperties']['source'])\n",
    "    print('~~~~~~~~~')\n",
    "    for name in result['contributors']:\n",
    "            print('{} - {}'.format(name['name'].encode('utf-8'), name.get('email')))\n",
    "    print('-------------------------------------------')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## No shared taxonomy for subjects or explicit document types - manuscripts, data, figures, etc.\n",
    "\n",
    "One of our developers is working on this right now!\n",
    "\n",
    "The DC field \"Type\" is an excellent place to start - however, there is a lot of variation for what is allowed inside of this field. \n",
    "\n",
    "\"Element Description: The nature or genre of the content of the resource. Type includes terms describing general categories, functions, genres, or aggregation levels for content. Recommended best practice is to select a value from a controlled vocabulary (for example, the DCMIType vocabulary ).\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sharepa import ShareSearch, basic_search\n",
    "from sharepa.helpers import pretty_print\n",
    "\n",
    "type_search = ShareSearch()\n",
    "total_documents = basic_search.count()\n",
    "\n",
    "type_search.aggs.bucket(\n",
    "    'typeTermFilter',  # Every aggregation needs a name\n",
    "    'terms',  # There are many kinds of aggregations\n",
    "    field='otherProperties.properties.type',\n",
    "    exclude= \"of|and|or\",\n",
    "    size=50,\n",
    ")\n",
    "\n",
    "type_results_executed = type_search.execute()\n",
    "\n",
    "type_results = type_results_executed.aggregations.typeTermFilter.to_dict()['buckets']\n",
    "\n",
    "type_dataframe = pd.DataFrame(type_results)\n",
    "type_dataframe['percent'] = (type_dataframe['doc_count'] / total_documents)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_count</th>\n",
       "      <th>key</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1328663</td>\n",
       "      <td>article</td>\n",
       "      <td>21.783339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1167877</td>\n",
       "      <td>text</td>\n",
       "      <td>19.147264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1066645</td>\n",
       "      <td>journal</td>\n",
       "      <td>17.487572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>195793</td>\n",
       "      <td>paper</td>\n",
       "      <td>3.210013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>189537</td>\n",
       "      <td>book</td>\n",
       "      <td>3.107446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>186536</td>\n",
       "      <td>figure</td>\n",
       "      <td>3.058245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>185722</td>\n",
       "      <td>dataset</td>\n",
       "      <td>3.044899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>117738</td>\n",
       "      <td>info:eu</td>\n",
       "      <td>1.930306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>117738</td>\n",
       "      <td>repo</td>\n",
       "      <td>1.930306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>117738</td>\n",
       "      <td>semantics</td>\n",
       "      <td>1.930306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>116310</td>\n",
       "      <td>chapter</td>\n",
       "      <td>1.906894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>109255</td>\n",
       "      <td>thesis</td>\n",
       "      <td>1.791228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>90199</td>\n",
       "      <td>masterthesis</td>\n",
       "      <td>1.478806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>42619</td>\n",
       "      <td>report</td>\n",
       "      <td>0.698736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>40624</td>\n",
       "      <td>conferenceobject</td>\n",
       "      <td>0.666028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>38477</td>\n",
       "      <td>proceedings</td>\n",
       "      <td>0.630828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>38257</td>\n",
       "      <td>peerreviewed</td>\n",
       "      <td>0.627221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>38242</td>\n",
       "      <td>component</td>\n",
       "      <td>0.626975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>29373</td>\n",
       "      <td>dissertation</td>\n",
       "      <td>0.481568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>29129</td>\n",
       "      <td>technical</td>\n",
       "      <td>0.477568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>29011</td>\n",
       "      <td>conference</td>\n",
       "      <td>0.475633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22989</td>\n",
       "      <td>fileset</td>\n",
       "      <td>0.376903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>20962</td>\n",
       "      <td>other</td>\n",
       "      <td>0.343671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>18077</td>\n",
       "      <td>journalarticle</td>\n",
       "      <td>0.296371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>18070</td>\n",
       "      <td>reference</td>\n",
       "      <td>0.296256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>18048</td>\n",
       "      <td>entry</td>\n",
       "      <td>0.295896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>17354</td>\n",
       "      <td>research</td>\n",
       "      <td>0.284518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>14891</td>\n",
       "      <td>type</td>\n",
       "      <td>0.244137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>14880</td>\n",
       "      <td>http</td>\n",
       "      <td>0.243957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>14879</td>\n",
       "      <td>eprint</td>\n",
       "      <td>0.243940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>14879</td>\n",
       "      <td>purl.org</td>\n",
       "      <td>0.243940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>13564</td>\n",
       "      <td>doctoralthesis</td>\n",
       "      <td>0.222381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>12170</td>\n",
       "      <td>texto</td>\n",
       "      <td>0.199526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>11928</td>\n",
       "      <td>publishedversion</td>\n",
       "      <td>0.195559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>10580</td>\n",
       "      <td>bookpart</td>\n",
       "      <td>0.173458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>10538</td>\n",
       "      <td>media</td>\n",
       "      <td>0.172770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>8955</td>\n",
       "      <td>sound</td>\n",
       "      <td>0.146817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>8834</td>\n",
       "      <td>performance</td>\n",
       "      <td>0.144833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>6261</td>\n",
       "      <td>nonpeerreviewed</td>\n",
       "      <td>0.102649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>6259</td>\n",
       "      <td>bachelorthesis</td>\n",
       "      <td>0.102616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>6187</td>\n",
       "      <td>survey</td>\n",
       "      <td>0.101435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>5835</td>\n",
       "      <td>document</td>\n",
       "      <td>0.095664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>5330</td>\n",
       "      <td>issue</td>\n",
       "      <td>0.087385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>4931</td>\n",
       "      <td>patent</td>\n",
       "      <td>0.080843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>4695</td>\n",
       "      <td>presentation</td>\n",
       "      <td>0.076974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>4557</td>\n",
       "      <td>version</td>\n",
       "      <td>0.074712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>4375</td>\n",
       "      <td>image</td>\n",
       "      <td>0.071728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>3789</td>\n",
       "      <td>data</td>\n",
       "      <td>0.062120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>3672</td>\n",
       "      <td>lecture</td>\n",
       "      <td>0.060202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>2891</td>\n",
       "      <td>still</td>\n",
       "      <td>0.047398</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    doc_count               key    percent\n",
       "0     1328663           article  21.783339\n",
       "1     1167877              text  19.147264\n",
       "2     1066645           journal  17.487572\n",
       "3      195793             paper   3.210013\n",
       "4      189537              book   3.107446\n",
       "5      186536            figure   3.058245\n",
       "6      185722           dataset   3.044899\n",
       "7      117738           info:eu   1.930306\n",
       "8      117738              repo   1.930306\n",
       "9      117738         semantics   1.930306\n",
       "10     116310           chapter   1.906894\n",
       "11     109255            thesis   1.791228\n",
       "12      90199      masterthesis   1.478806\n",
       "13      42619            report   0.698736\n",
       "14      40624  conferenceobject   0.666028\n",
       "15      38477       proceedings   0.630828\n",
       "16      38257      peerreviewed   0.627221\n",
       "17      38242         component   0.626975\n",
       "18      29373      dissertation   0.481568\n",
       "19      29129         technical   0.477568\n",
       "20      29011        conference   0.475633\n",
       "21      22989           fileset   0.376903\n",
       "22      20962             other   0.343671\n",
       "23      18077    journalarticle   0.296371\n",
       "24      18070         reference   0.296256\n",
       "25      18048             entry   0.295896\n",
       "26      17354          research   0.284518\n",
       "27      14891              type   0.244137\n",
       "28      14880              http   0.243957\n",
       "29      14879            eprint   0.243940\n",
       "30      14879          purl.org   0.243940\n",
       "31      13564    doctoralthesis   0.222381\n",
       "32      12170             texto   0.199526\n",
       "33      11928  publishedversion   0.195559\n",
       "34      10580          bookpart   0.173458\n",
       "35      10538             media   0.172770\n",
       "36       8955             sound   0.146817\n",
       "37       8834       performance   0.144833\n",
       "38       6261   nonpeerreviewed   0.102649\n",
       "39       6259    bachelorthesis   0.102616\n",
       "40       6187            survey   0.101435\n",
       "41       5835          document   0.095664\n",
       "42       5330             issue   0.087385\n",
       "43       4931            patent   0.080843\n",
       "44       4695      presentation   0.076974\n",
       "45       4557           version   0.074712\n",
       "46       4375             image   0.071728\n",
       "47       3789              data   0.062120\n",
       "48       3672           lecture   0.060202\n",
       "49       2891             still   0.047398"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Documents with Descriptions\n",
    "\n",
    "Are abstracts copyrightable?\n",
    "\n",
    "Abstracts, summaries and descriptions are not always made available. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-13-ce941d22f81c>, line 6)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-13-ce941d22f81c>\"\u001b[0;36m, line \u001b[0;32m6\u001b[0m\n\u001b[0;31m    percent = results['count']/total_documents)*100\u001b[0m\n\u001b[0m                                              ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# How many documents have descriptions?\n",
    "from __future__ import division\n",
    "\n",
    "results = query_share(10, 'description:*')\n",
    "\n",
    "percent = results['count']/total_documents)*100\n",
    "\n",
    "print('There are {} results'.format(results['count']))\n",
    "print('{}/{} or {}% of results have descriptions'.format(\n",
    "        results['count'], total_documents,\n",
    "        format(percent, '.2f')\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}