text_mining_tutorial_using_Entrez.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Mining NCBI Pubmed\n",
    "\n",
    "This session will cover the use of the Entrez module located within the popular package BioPython. You will learn how to query the NCBI's Pubmed database from a Python3 script (e.g. the Jupyter notebook herein) using key terms before filtering the results to find relevant literature for a given investigation. Although not explored here, the Entrez package works well alongside the natural language processing module, NLTK. Therefore, this session is designed to introduce the novice Python programmer to a few of the initial steps required for a project involving literature searches, natural language processing and/or machine learning.\n",
    "\n",
    "Before starting this session, you will need to be familiar with Pandas and Matplotlib. Take a look at our previous sessions on Python for the Central Dogma and An Introduction to Pandas/Matplotlib to get yourself up to speed.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Let's start by importing the Entrez module from BioPython. Entrez will sometimes return an error message if you do not specify your ```email``` but it will not affect how your code runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from Bio import Entrez\n",
    "\n",
    "Entrez.email = 'example_name@alphabetmail.com'  # Optional. Typically for large-scale searches involving thousands of queries\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line of code below will use Entrez to search for the five most recent entries in the NCBI Pubmed (```pubmed```) database that cite the term ```bioinformatics``` somewhere in the manuscript.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = Entrez.esearch(db='pubmed',            # Sets NCBI Pubmed as the database\n",
    "                        term='bioinformatics',  # Searches for papers including the term 'bioinformatics'\n",
    "                        retmax='5'              # Sets the maximum number of hits returned\n",
    "                       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our ```search``` variable is currently stored as a class object somewhere in your computer's memory and simply printing it will display a cryptic line of alphanumeric characters. Try it yourself; use the ```print()``` function on the class variable ```search```. \n",
    "\n",
    "We, therefore, need to run the ```read()``` function on our ```search``` variable, which can be done using the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_results = Entrez.read(search)  # Returns a dictionary of data from the search\n",
    "\n",
    "print(search_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of printing the ```search``` results using ```read()``` is a dictionary containing metadata on your query. ```Count``` is the number of documented papers containing your search term(s), ```IdList``` is the list of the Pubmed IDs returned from the search, and so on. For a fuller description of the output keys from Entrez, refer to [the documentation](https://biopython.readthedocs.io/en/latest/api/Bio.Entrez.html). \n",
    "\n",
    "---\n",
    "\n",
    "We will continue this session by looking at the Pubmed IDs from our search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(search_results['IdList'])   # Prints the Pubmed IDs from search. Set retmax to a higher integer to get more results.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entrez has a useful function called ```efetch()``` that takes a Pubmed ID and returns data the paper it refers to. \n",
    "\n",
    "Let's try this using a loop and our five Pubmed IDs. Depending on your internet speed, this can take a few seconds to minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "literature_list = []     # Initialise empty list to store abstracts\n",
    "\n",
    "for entry in search_results['IdList']:\n",
    "    \n",
    "    entry_details = Entrez.efetch(id=entry,             # Pubmed ID\n",
    "                                  db='pubmed',          # Database\n",
    "                                  rettype='Abstract',   # Return type = abstract. Refer to Entrez documentation for other rettypes\n",
    "                                  retmode='text')       # Return output as human-readable text\n",
    "    \n",
    "    literature_list.append(entry_details.read().lower())    # Adds fetched abstracts to list\n",
    "                                                            # Converted to lower case\n",
    "print(literature_list)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a list of paper abstracts citing the term bioinformatics somewhere in the text (not necessarily in the abstract). \n",
    "\n",
    "## Task 1\n",
    "\n",
    "Iterate through this list, print out each paper's abstract and its index in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 1 ### \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the output of the code above, ```Entrez.efetch()``` supplies metadata and author information along with the paper's abstract. For the purpose of this session (and many other applications of text mining), including this information in tandem with the abstract's text does not much matter. \n",
    "\n",
    "---\n",
    "\n",
    "Excellent! We now have a list of abstracts all somehow related to bioinformatics. Let's explore our results a little further. \n",
    "\n",
    "## Task 2\n",
    "\n",
    "Write some code in the block below to print the number of occurrences of the word \"gene\" in each paper's abstract. What is the index of the paper with the highest number of occurrences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Tasks 2 ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Simply finding the word count for particular terms in a paper's abstract can often be an effective way to quickly guess the context of a paper and assess its relevance to the study. For example, a paper prevalently containing the word \"gene\" in its abstract after a search for the term \"bioinformatics\" could suggest the paper reports a genomics study. Finer context can be assessed by finding the counts for more words within the abstract. \n",
    "\n",
    "## Task 3\n",
    "\n",
    "By using a Python dictionary, find the frequency occurrence for the terms ```gene```, ```protein``` and ```RNA``` in each abstact and print the results. What does this tell you about the first, fourth and fifth abstracts? How might you use this to narrow down a literature search?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 3 ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another operation we might want to perform is finding the total counts for certain key terms over all of our reported abstracts. In this case, let's find how prevalent the key terms ```gene```, ```protein``` and ```RNA``` occur in our abstracts from papers citing the term 'bioinformatics'. Our example of five papers makes this task trivial - we could simply add up the counts from task 3 - but scaling the literature search up to hundreds or thousands of papers quickly becomes cumbersome. \n",
    "\n",
    "## Task 4\n",
    "\n",
    "Modify the code above to find the total frequency of the terms ```gene```, ```protein``` and ```RNA``` across all abstracts in the ```literature_list```. What could this tell you about published papers citing the term 'bioinformatics'?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 4 ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Task 5\n",
    "\n",
    "Based on some of the examples from the previous session, plot an appropriate graph for this data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 5 ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Task 6\n",
    "\n",
    "Using all the necessary tools we have covered in this session, produce a barchart for the frequency of the terms in the list below from the abstracts of the latest 50 publications that contain the term 'systems biology' in the NCBI Pubmed database. \n",
    "\n",
    "Terms to look for in the abstract:\n",
    "\n",
    "- gene\n",
    "- protein\n",
    "- rna\n",
    "- metabolite\n",
    "- genomic\n",
    "- proteomic\n",
    "- transcriptomic\n",
    "- metabolomic\n",
    "\n",
    "Note: The ```term``` argument in Entrez's ```esearch()``` function requires spaces to be written with the ```+``` symbol (e.g. ```term='Hello+wolrd``` means search for the term ```Hello world```).\n",
    "\n",
    "Fetching the abstracts from all 50 queries might take several minutes. Therefore, suitably reduce this number if you have either slow internet or need the results quickly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 6 ###\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}