Webinar_April_2021.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GDC April 2021 Webinar: Using the GDC API\n",
    "\n",
    "### Monday, April 26, 2021<br>2:00 PM - 3:00 PM (EST)<br>Bill Wysocki, Director of User Services <br>University of Chicago\n",
    "\n",
    "## <a id='toc'>Table of Contents</a>\n",
    "\n",
    "- [API User's Guide and Other Helpful Links](#links)\n",
    "- [Notebook Overview](#overview)\n",
    "    - [About this notebook](#about_notebook)\n",
    "    - [Using the Python requests package and interpreting request reponse messages](#requests_package)\n",
    "- [GDC API Overview](#api_overview)\n",
    "    - [GDC API Format](#api_format)\n",
    "    \n",
    "- [Using the GDC API to Query Data in GDC](#query_data)\n",
    "    - [Search and Retrieval Endpoints Examples](#search_retrieve)\n",
    "    - [Data Analysis Endpoints Examples](#analysis)\n",
    "- [Using the GDC API to Submit Data to GDC](#submit)\n",
    "\n",
    "## <a id='links'>API User's Guide and Other Helpful Links</a>\n",
    "\n",
    "[GDC API User's Guide](https://docs.gdc.cancer.gov/API/Users_Guide/Getting_Started/)\n",
    "\n",
    "[GDC Support Website](https://gdc.cancer.gov/support)\n",
    "\n",
    "support@nci-gdc.datacommons.io - GDC Helpdesk E-mail\n",
    "\n",
    "[Requests Python Package User's Guide](https://2.python-requests.org/en/master/)\n",
    "\n",
    "[Python Documentation Website](https://www.python.org)\n",
    "\n",
    "[Jupyter Notebook Documentation](https://jupyter.org/documentation)\n",
    "\n",
    "# <a id='overview'>Notebook Overview</a>\n",
    "\n",
    "\n",
    "### <a id='about_notebook'>About this notebook</a>\n",
    "\n",
    "- This notebook serves to be a resource for GDC users to familiarize themselves with GDC API endpoints and allow users to edit and create custom queries \"in-place\" with provided template functions or submission tasks\n",
    "- The provided functional templates can facilitate downstream data analyses and visualizations within the Jupyter Notebook interface and other Python packages\n",
    "- Commands and functions in this notebook will rely on the following Python packages:\n",
    "    - `requests` - if not already installed on your system, can install with command `pip install requests` from command line or using a new code cell in this notebook\n",
    "    - `json` - part of Python standard library, should already be installed on system\n",
    "    - `urllib` - part of Python standard library, should already be installed on system\n",
    "- To execute code in a code cell, press either 'Cmd + Enter' or 'Control + Enter' depending on operating system and keyboard layout\n",
    "- If using notebook to aid in submission requests, will need to download token file from the [GDC Submission Portal](https://docs.gdc.cancer.gov/Data_Submission_Portal/Users_Guide/Data_Submission_Process/#authentication)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import packages to use in this notebook\n",
    "\n",
    "import requests\n",
    "import json\n",
    "import urllib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='requests_package'>Using the Python `requests` package and interpreting request reponse messages</a>\n",
    "\n",
    "- The `requests` package allows users to communicate with the GDC API to make standard `POST`, `PUT`, `GET` and `DELETE` HTTP methods\n",
    "-  Need to specify request method as part of function (i.e. `request.get()` for `GET` method, `request.post()` for `POST` method etc.)\n",
    "- When making a request with `requests` package, can save results of request as variable, i.e.:\n",
    "    - `response = requests.get(url)`\n",
    "- Example `GET` request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get('https://api.gdc.cancer.gov/cases?filters=%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cases.project.program.name%22%2C%20%22value%22%3A%20%22TCGA%22%7D%7D&fields=submitter_slide_ids&size=1&format=json&pretty=true')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When returning the contents of the `response` variable, will only return HTTP status code of request, such as `<Response 200>` or `<Response 400>`; need to specify `response.text` method to get return message or data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': {'hits': [{'id': '375436b3-66ac-4d5e-b495-18a96d812a69',\n",
       "    'submitter_slide_ids': ['TCGA-F5-6810-01Z-00-DX1',\n",
       "     'TCGA-F5-6810-01A-01-TS1',\n",
       "     'TCGA-F5-6810-01A-01-BS1']}],\n",
       "  'pagination': {'count': 1,\n",
       "   'total': 11315,\n",
       "   'size': 1,\n",
       "   'from': 0,\n",
       "   'sort': '',\n",
       "   'page': 1,\n",
       "   'pages': 11315}},\n",
       " 'warnings': {}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.loads(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Typically, successful responses begin with `'2'`, like `200` or `201` and unsuccessful requests begin with `'4'`, like `400` (bad request) or `403` ('forbidden' error, result of bad or insufficient credentials)\n",
    "- A list and accompanying explanations of HTTP status codes can be [found here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='api_overview'>GDC API Overview</a>\n",
    "\n",
    "\n",
    "- The GDC Application Programming Interface (API) is the external facing REpresentational State Transfer (REST) interface for the GDC\n",
    "- The GDC API supports user interactions with the GDC Submission and Data Portals, as well as provides developers with a programmatic interface to query and download GDC data, metadata and annotations and submit data to the GDC.\n",
    "- The [GDC Data Transfer Tool](https://gdc.cancer.gov/access-data/gdc-data-transfer-tool) client also relies on the GDC API for user authentication, reading manifests, and for download and upload features\n",
    "\n",
    "\n",
    "### <a id='api_format'>GDC API Format</a>\n",
    "\n",
    "- The HTTP URL that corresponds to the GDC API is: https://api.gdc.cancer.gov/\n",
    "- GDC API format for search and retrieval use is: <b>API_URL + ENDPOINT + QUERY_PARAMETERS</b>\n",
    "- In order to utilize the GDC API, calls to specific API 'endpoints' for a given query need to be made, i.e. for retreiving data about cases in the GDC, will make calls to `cases` endpoint, https://api.gdc.cancer.gov/cases/\n",
    "- For search and retrieval API calls, query parameters can be included, such as <b>filters</b> on endpoint fields, and the <b>fields</b> parameter to specify fields to return from query\n",
    "    - List of all indexed data fields to use specify as filters or fields for search and retrieval endpoints can be found at https://docs.gdc.cancer.gov/API/Users_Guide/Appendix_A_Available_Fields/\n",
    "    - Can also view available fields for both Search and Retrieval and Data Analysis endpoints by [using the `_mapping` endpoint for a given endpoint](https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#_mapping-endpoint) or at the corresponding pages at the [GDC API Documentation site](https://docs.gdc.cancer.gov/API/Users_Guide/Getting_Started/)\n",
    "    - Formatting parameters can be specified such as <b>format</b> (TSV or JSON format) and <b>size</b> (number of hits to return)\n",
    "- For submitting data using the GDC API, the format for using the GDC API Submission endpoint uses the project ID: https://api.gdc.cancer.gov/<b>program_name/project_code</b>, i.e. https://api.gdc.cancer.gov/submission/TCGA/LUAD or https://api.gdc.cancer.gov/submission/CPTAC/3 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='query_data'>Using the GDC API to Query Data in GDC</a>\n",
    "\n",
    "### Overview\n",
    "\n",
    "- Submitters can make use of several GDC API endpoints to retrieve various data indexed in the GDC API, including biospecimen, clinical and annotation metadata\n",
    "- The HTTP `GET` method will be used to retrieve data\n",
    "- Additional parameters can be specified to tailor the returned data, such as number of returned entries and filters on data at endpoint\n",
    "- Data can be retrieved in `JSON` or `TSV` format by specifying in the request the format desired (see below)\n",
    "- Additional features and more information regarding using the GDC API can be found at this link: https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/\n",
    "\n",
    "\n",
    "\n",
    "### Endpoints\n",
    "\n",
    "There are two 'types' of endpoints that can be used to query data in the GDC:\n",
    "\n",
    "\n",
    "[GDC Search and Retrieval Endpoints](https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#endpoints) - includes endpoints that index project, file and case information, including clinical and biospecimen metadata, as well as file version and history\n",
    "\n",
    "[GDC Analysis Endpoints](https://docs.gdc.cancer.gov/API/Users_Guide/Data_Analysis/) - endpoints that are used by the GDC data analysis, visualization and exploration (DAVE) tools in the Exploration tab of the GDC Data Portal to access indexed data including gene, mutation, copy number variation and survival data. \n",
    "\n",
    "\n",
    "### Steps\n",
    "\n",
    "1. Specify and percent-encode `filters`\n",
    "2. Specify `fields` to be returned\n",
    "3. Specify additional parameters (`size`, `format` of results etc.)\n",
    "3. Concatenate parameters to build query url\n",
    "4. Submit query and save response text to file\n",
    "\n",
    "Note: specifying parameters are optional; not specifying `filters` will return all instances at a given endpoint, and not specifying `fields` will return all fields at endpoint, while other parameters will be set to default value (i.e. `size` = 10, `format` = JSON)\n",
    "\n",
    "### Template queryBuilder() function\n",
    "\n",
    "- `GET` requests can be built as a URL with the endpoint and other parameters specified using a Python function\n",
    "- In notebook, need to first run code for queryBuilder() function to instantiate the function\n",
    "- Parameters must be passed into parantheses in the order that they are specified in the function\n",
    "- To specify default parameters, users can simply input two quotation marks, i.e. `''`, for a given variable when using the queryBuilder() function\n",
    "- Users can edit the template queryBuilder() function to build url request for querying data in GDC API to include other parameters, such as `facets`, `expand`, `from` (pagination) and `sort`: \n",
    "https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#request-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#format is specified as 'frmat' in function as format is an already declared object in python [the format() function]\n",
    "\n",
    "def queryBuilder(endpoint, filters, fields, size, frmat):\n",
    "    api_url = 'https://api.gdc.cancer.gov/'\n",
    "    \n",
    "    if frmat.lower() == 'json':\n",
    "        request_query = api_url + endpoint + '?filters=' + filters + '&fields=' + fields + '&size=' + size + '&format=' + frmat + '&pretty=true'\n",
    "    else:\n",
    "        request_query = api_url + endpoint + '?filters=' + filters + '&fields=' + fields + '&size=' + size + '&format=' + frmat\n",
    "    return request_query"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Templates for query `filters`\n",
    "\n",
    "- `Filters` are used to specify which hits to return from an endpoint, such as cases of a certain project or files from a certain workflow\n",
    "- Filters need to be created in JSON format that then will need to be [percent-encoded]() to be sent in the URL request (can use the `urllib` Python package for percent-endcoded formatting)\n",
    "- JSON filters use [operators](https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#filters-specifying-the-query) to specify relationships between a field and their possible values\n",
    "- For a given endpoint, need to use indexed fields at that endpoint\n",
    "    - For Search and Retrieval endpoints, can reference [Appendix A at GDC API Documentation site](https://docs.gdc.cancer.gov/API/Users_Guide/Appendix_A_Available_Fields/)\n",
    "    - Can also view available fields for both Search and Retrieval and Data Analysis endpoints by [using the `_mapping` endpoint for a given endpoint](https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#_mapping-endpoint)\n",
    "- Specifying no filters will return all instances for a given endpoint (default)\n",
    "- Below are several examples users can edit to build filters for a `GET` request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#one filter applied to endpoint\n",
    "\n",
    "#one filter \n",
    "one_filter = {\n",
    "            \"op\":\"=\",\n",
    "            \"content\":{\n",
    "                \"field\": \"cases.project.project_id\", \n",
    "                \"value\": \"TCGA-BRCA\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#combination of two filters applied to endpoint, i.e. (x AND/OR y) must be met\n",
    "\n",
    "combination_two = {\n",
    "    \"op\" : \"and\",\n",
    "    \"content\":[{\n",
    "        \"op\":\"=\",\n",
    "         \"content\":{\n",
    "              \"field\": \"cases.project.project_id\", \n",
    "                \"value\": \"TCGA-BRCA\"\n",
    "            }\n",
    "        }, \n",
    "        {\n",
    "            \"op\":\"=\", \n",
    "            \"content\":{\n",
    "                \"field\":\"cases.disease_type\",\n",
    "                \"value\": \"ductal and lobular neoplasms\"\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#combination of three filters applied to endpoint, i.e. (x AND/OR y AND/OR z) must be met\n",
    "\n",
    "combination_three = {\n",
    "    \"op\" : \"and\",\n",
    "    \"content\":[{\n",
    "        \"op\":\"=\",\n",
    "         \"content\":{\n",
    "              \"field\": \"cases.project.project_id\", \n",
    "                \"value\": \"TCGA-BRCA\"\n",
    "            }\n",
    "        }, \n",
    "        {\n",
    "            \"op\":\"=\", \n",
    "            \"content\":{\n",
    "                \"field\":\"cases.disease_type\",\n",
    "                \"value\": \"ductal and lobular neoplasms\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"op\":\">\", \n",
    "            \"content\":{\n",
    "                \"field\":\"diagnoses.age_at_diagnosis\",\n",
    "                \"value\": \"15000\"\n",
    "            }\n",
    "        }\n",
    "        \n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#complex combination of three filters applied to endpoint, i.e. (x AND/OR [y AND/OR z]) must be met\n",
    "\n",
    "combination_three_2 = {\n",
    "    \"op\": \"and\",\n",
    "    \"content\": [{\n",
    "            \"op\": \"=\",\n",
    "            \"content\": {\n",
    "                \"field\": \"cases.project.project_id\",\n",
    "                \"value\": \"TCGA-BRCA\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"op\": \"or\",\n",
    "            \"content\": [{\n",
    "                    \"op\": \"=\",\n",
    "                    \"content\": {\n",
    "                        \"field\": \"cases.disease_type\",\n",
    "                        \"value\": \"cystic, mucinous and serious neoplasms\"\n",
    "                    }\n",
    "                },\n",
    "                {\n",
    "                    \"op\": \"=\",\n",
    "                    \"content\": {\n",
    "                        \"field\": \"cases.disease_type\",\n",
    "                        \"value\": \"ductal and lobular neoplasms\"\n",
    "                    }\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Template commands for formatting filter parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#percent encoding of filters\n",
    "json_string=str(json.dumps(one_filter)) #replace one_filter with input filter variable here\n",
    "example_filter = urllib.parse.quote(json_string.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Template for formatting `fields` to be returned by query\n",
    "\n",
    "- The `fields` parameter is passed to the API request URL as a comma-delimited list of fields to be returned\n",
    "- For a given endpoint, can only specify indexed fields at that endpoint\n",
    "    - For Search and Retrieval endpoints, can reference [Appendix A at GDC API Documentation site](https://docs.gdc.cancer.gov/API/Users_Guide/Appendix_A_Available_Fields/)\n",
    "    - Can also view available fields for both Search and Retrieval and Data Analysis endpoints by [using the `_mapping` endpoint for a given endpoint](https://docs.gdc.cancer.gov/API/Users_Guide/Search_and_Retrieval/#_mapping-endpoint)\n",
    "- Specifying no fields will return all available fields for entities that match `filters` for a given endpoint (default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#specify fields to be returned\n",
    "example_fields = \",\".join([\n",
    "    \"submitter_id\",\n",
    "    \"disease_type\",\n",
    "    \"samples.submitter_id\",\n",
    "    \"samples.sample_type\", \n",
    "    \"samples.tissue_type\",\n",
    "    \"diagnoses.age_at_diagnosis\"\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Template API `GET` Request "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#build API query: queryBuilder(endpoint, filters, fields, size, frmat)\n",
    "\n",
    "#to specify no filters and/or no fields to return, replace variable with ''\n",
    "\n",
    "template_request = queryBuilder('cases', example_filter, example_fields, '11315', \"json\")\n",
    "\n",
    "template_request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### <font color=\"red\">Note: You can also copy and paste formatted request URL into browser url bar to  return results in browser</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#send request\n",
    "result = requests.get(template_request)\n",
    "\n",
    "#write request results to file, edit file name and type \n",
    "with open(\"/Users/catherineausland/Desktop/ffpe.json\", \"w+\") as output: \n",
    "    output.write(result.text)\n",
    "output.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='search_retrieve'>Search and Retrieval Endpoints Examples</a>\n",
    "\n",
    "### Example 1: Retrieve case barcode, sample type and primary diagnosis data for DNA-seq files in TCGA-BRCA project\n",
    "\n",
    "- For this example, we would like to retrieve whether BAM files in the TCGA-BRCA project are for normal or tumor samples, as well as what disease cases were diagnosed as\n",
    "- Use 'files' endpoint, as this endpoint contains metadata related to files in the GDC (such as experimental strategy and data category)\n",
    "- Need to filter down to files that are of the data category \"sequencing reads\" and experimental strategy type \"WXS\" (whole exome) to filter out other categories (like copy number variation, gene expression) and other experimental stragies (like RNA-Seq). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#step 1: specify and encode filters\n",
    "\n",
    "filters = {\n",
    "    \"op\" : \"and\",\n",
    "    \"content\":[{\n",
    "        \"op\":\"=\",\n",
    "         \"content\":{\n",
    "              \"field\": \"cases.project.project_id\", \n",
    "                \"value\": \"TCGA-BRCA\"\n",
    "            }\n",
    "        }, \n",
    "        {\n",
    "            \"op\":\"=\", \n",
    "            \"content\":{\n",
    "                \"field\":\"files.data_category\",\n",
    "                \"value\": \"sequencing reads\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"op\":\"=\", \n",
    "            \"content\":{\n",
    "                \"field\":\"files.experimental_strategy\",\n",
    "                \"value\": \"WXS\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"op\":\"=\", \n",
    "            \"content\":{\n",
    "                \"field\":\"files.data_format\",\n",
    "                \"value\": \"BAM\"\n",
    "            }\n",
    "        }\n",
    "        \n",
    "    ]\n",
    "}\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify fields to be returned\n",
    "fields = \",\".join([\n",
    "    \"cases.submitter_id\",\n",
    "    \"file_name\",\n",
    "    \"cases.samples.sample_type\",\n",
    "    \"cases.diagnoses.primary_diagnosis\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=1 and format=tsv, build query url with 'files' endpoint\n",
    "brca_request = queryBuilder('files', filters_format, fields, '1', \"tsv\")\n",
    "\n",
    "#step 5: send request\n",
    "brca_result = requests.get(brca_request)\n",
    "\n",
    "print(brca_result.text)\n",
    "brca_request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Retrieve FFPE data for samples and portions for TCGA projects\n",
    "\n",
    "- In this example, we will retrieve whether case samples and portions taken from cases in TCGA projects were Formalin-Fixed Paraffin-Embedded (FFPE) specimens or not\n",
    "- Use the 'cases' endpoint, as this endpoint contains biospecien and clinical information related to cases and samples in the GDC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://api.gdc.cancer.gov/cases?filters=%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cases.project.program.name%22%2C%20%22value%22%3A%20%22TCGA%22%7D%7D&fields=submitter_id,samples.submitter_id,samples.is_ffpe,samples.portions.submitter_id,samples.portions.is_ffpe&size=1&format=json&pretty=true'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#step 1: specify and encode filters\n",
    "filters = {\n",
    "            \"op\":\"=\",\n",
    "            \"content\":{\n",
    "                \"field\": \"cases.project.program.name\", \n",
    "                \"value\": \"TCGA\"\n",
    "    }\n",
    "}\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify fields to be returned\n",
    "fields = \",\".join([\n",
    "    \"submitter_id\",\n",
    "    \"samples.submitter_id\",\n",
    "    \"samples.is_ffpe\",\n",
    "    \"samples.portions.submitter_id\",\n",
    "    \"samples.portions.is_ffpe\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=1 and format=json, build query url with 'cases' endpoint\n",
    "ffpe_request = queryBuilder('cases', filters_format, fields, '1', \"json\")\n",
    "\n",
    "#step 5: send request\n",
    "ffpe_result = requests.get(ffpe_request)\n",
    "\n",
    "print(ffpe_result.text)\n",
    "ffpe_request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Age at Diagnosis, Days to Death after Diagnosis, Vital Status and other clinical data for cases in TCGA-KIRC project\n",
    "\n",
    "- In this example, we will retrieve age, survival and other clinical data for cases in the TCGA-KIRC project\n",
    "- Use the 'cases' endpoint, as this endpoint contains biospecien and clinical information related to cases and samples in the GDC\n",
    "- Results will only show data for `demographic.days_to_death` if case is deceased"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "demographic.days_to_death\tdemographic.ethnicity\tdemographic.gender\tdemographic.race\tdemographic.vital_status\tdiagnoses.0.age_at_diagnosis\tid\tsubmitter_id\r\n",
      "\tnot hispanic or latino\tmale\twhite\tAlive\t15197\t24c1cf70-bd67-431e-a623-d20f8d3f52b2\tTCGA-CJ-4881\r\n",
      "866\tnot hispanic or latino\tmale\twhite\tDead\t21317\t12adefc4-c9fd-46d3-904b-8fc52d5f1913\tTCGA-B0-4696\r\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://api.gdc.cancer.gov/cases?filters=%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cases.project.project_id%22%2C%20%22value%22%3A%20%22TCGA-KIRC%22%7D%7D&fields=submitter_id,diagnoses.age_at_diagnosis,demographic.days_to_death,demographic.vital_status,demographic.ethnicity,demographic.race,demographic.gender&size=2&format=tsv'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#step 1: specify and encode filters\n",
    "filters = {\n",
    "            \"op\":\"=\",\n",
    "            \"content\":{\n",
    "                \"field\": \"cases.project.project_id\", \n",
    "                \"value\": \"TCGA-KIRC\"\n",
    "    }\n",
    "}\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify fields to be returned\n",
    "fields = \",\".join([\n",
    "    \"submitter_id\",\n",
    "    \"diagnoses.age_at_diagnosis\",\n",
    "    \"demographic.days_to_death\",\n",
    "    \"demographic.vital_status\", \n",
    "    \"demographic.ethnicity\",\n",
    "    \"demographic.race\",\n",
    "    \"demographic.gender\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=2 and format=tsv, build query url with 'cases' endpoint\n",
    "age_request = queryBuilder('cases', filters_format, fields, '2', \"tsv\")\n",
    "\n",
    "#step 5: send request\n",
    "age_result = requests.get(age_request)\n",
    "\n",
    "print(age_result.text)\n",
    "age_request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='analysis'>Data Analysis Endpoints Examples</a>\n",
    "\n",
    "### Example 4: Gene information\n",
    "\n",
    "- In this example, we will retrieve gene IDs and positions of genes present on chromosome 8 of the human genome\n",
    "- Use the 'genes' endpoint, as this endpoint contains gene information indexed in the GDC API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gene_end\tgene_start\tid\tsymbol\r\n",
      "144511213\t144509074\tENSG00000167700\tMFSD3\r\n",
      "144502121\t144477969\tENSG00000160972\tPPP1R16A\r\n",
      "23083619\t23068229\tENSG00000246130\tRP11-875O11.2\r\n",
      "101669726\t101492432\tENSG00000083307\tGRHL2\r\n",
      "143609773\t143603913\tENSG00000104524\tPYCRL\r\n",
      "108787615\t108606850\tENSG00000164841\tTMEM74\r\n",
      "6759666\t6708357\tENSG00000155189\tAGPAT5\r\n",
      "22575788\t22544986\tENSG00000120896\tSORBS3\r\n",
      "43303482\t43303376\tENSG00000238509\tRNU6-104P\r\n",
      "97149654\t96645227\tENSG00000104324\tCPQ\r\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://api.gdc.cancer.gov/genes?filters=%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22gene_chromosome%22%2C%20%22value%22%3A%20%228%22%7D%7D&fields=id,symbol,gene_start,gene_end&size=10&format=tsv'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#step 1: specify and encode filters\n",
    "filters = {\n",
    "            \"op\":\"=\",\n",
    "            \"content\":{\n",
    "                \"field\": \"gene_chromosome\", \n",
    "                \"value\": \"8\"\n",
    "    }\n",
    "}\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify fields to be returned\n",
    "fields = \",\".join([\n",
    "    \"id\",\n",
    "    \"symbol\",\n",
    "    \"gene_start\",\n",
    "    \"gene_end\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=10 and format=tsv, build query url with 'genes' endpoint\n",
    "genes_request = queryBuilder('genes', filters_format, fields, '10', \"tsv\")\n",
    "\n",
    "#step 5: send request\n",
    "genes_result = requests.get(genes_request)\n",
    "\n",
    "print(genes_result.text)\n",
    "genes_request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data\": {\n",
      "    \"biotype\": \"protein_coding\", \n",
      "    \"symbol\": \"VPS28\", \n",
      "    \"canonical_transcript_length_genomic\": 4963, \n",
      "    \"cytoband\": [\n",
      "      \"8q24.3\"\n",
      "    ], \n",
      "    \"canonical_transcript_length\": 1097, \n",
      "    \"synonyms\": [], \n",
      "    \"description\": \"This gene encodes a protein subunit of the ESCRT-I complex (endosomal complexes required for transport), which functions in the transport and sorting of proteins into subcellular vesicles. This complex can also be hijacked to facilitate the budding of enveloped viruses from the cell membrane. Alternative splicing results in multiple transcript variants encoding different isoforms. [provided by RefSeq, Jul 2013]\", \n",
      "    \"canonical_transcript_length_cds\": 699, \n",
      "    \"gene_chromosome\": \"8\", \n",
      "    \"gene_end\": 144428563, \n",
      "    \"canonical_transcript_id\": \"ENST00000377348\", \n",
      "    \"gene_strand\": -1, \n",
      "    \"gene_start\": 144423601, \n",
      "    \"name\": \"vacuolar protein sorting 28 homolog (S. cerevisiae)\", \n",
      "    \"gene_id\": \"ENSG00000160948\"\n",
      "  }, \n",
      "  \"warnings\": {}\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "#Can use gene_id to also query individual information about the gene in question from genes endpoint as well\n",
    "#by appending the gene_id at the end of the 'genes' endpoint and specifying parameters\n",
    "\n",
    "individual_gene_request = requests.get('https://api.gdc.cancer.gov/genes/ENSG00000160948?pretty=true')\n",
    "\n",
    "print(individual_gene_request.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 5: Simple Somatic Mutation Information \n",
    "\n",
    "- In this example, we will retrieve information on a specific mutation using its COSMIC ID\n",
    "- Use the 'ssms' endpoint, as this endpoint contains mutation information indexed in the GDC API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data\": {\n",
      "    \"hits\": [\n",
      "      {\n",
      "        \"id\": \"8b3c1a7a-e4e0-5200-9d46-5767c2982145\", \n",
      "        \"start_position\": 62438203, \n",
      "        \"gene_aa_change\": [\n",
      "          \"CADPS G1147G\", \n",
      "          \"CADPS G1187G\", \n",
      "          \"CADPS G1217G\", \n",
      "          \"CADPS G1226G\", \n",
      "          \"CADPS G127G\", \n",
      "          \"CADPS G218G\", \n",
      "          \"CADPS G95G\"\n",
      "        ], \n",
      "        \"reference_allele\": \"C\", \n",
      "        \"ncbi_build\": \"GRCh38\", \n",
      "        \"cosmic_id\": [\n",
      "          \"COSM4860838\", \n",
      "          \"COSM731764\", \n",
      "          \"COSM731765\"\n",
      "        ], \n",
      "        \"mutation_subtype\": \"Single base substitution\", \n",
      "        \"mutation_type\": \"Simple Somatic Mutation\", \n",
      "        \"chromosome\": \"chr3\", \n",
      "        \"ssm_id\": \"8b3c1a7a-e4e0-5200-9d46-5767c2982145\", \n",
      "        \"genomic_dna_change\": \"chr3:g.62438203C>T\", \n",
      "        \"tumor_allele\": \"T\", \n",
      "        \"end_position\": 62438203\n",
      "      }\n",
      "    ], \n",
      "    \"pagination\": {\n",
      "      \"count\": 1, \n",
      "      \"total\": 1, \n",
      "      \"size\": 1, \n",
      "      \"from\": 0, \n",
      "      \"sort\": \"\", \n",
      "      \"page\": 1, \n",
      "      \"pages\": 1\n",
      "    }\n",
      "  }, \n",
      "  \"warnings\": {\n",
      "    \"fields\": \"unrecognized values: []\"\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://api.gdc.cancer.gov/ssms?filters=%7B%22op%22%3A%20%22in%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cosmic_id%22%2C%20%22value%22%3A%20%5B%22COSM4860838%22%5D%7D%7D&fields=&size=1&format=json&pretty=true'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#step 1: specify and encode filters\n",
    "filters =  {\n",
    "   \"op\":\"in\",\n",
    "   \"content\":{\n",
    "      \"field\":\"cosmic_id\",\n",
    "      \"value\":[\n",
    "         \"COSM4860838\"\n",
    "      ]\n",
    "   }\n",
    "}\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify all fields to be returned (default =  \"\")\n",
    "fields = \",\".join([\n",
    "    \"\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=1 and format=json, build query url with 'ssms' endpoint\n",
    "mutation_request = queryBuilder('ssms', filters_format, fields, '1', \"json\")\n",
    "\n",
    "#step 5: send request\n",
    "mutation_result = requests.get(mutation_request)\n",
    "\n",
    "print(mutation_result.text)\n",
    "mutation_request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 6: Compare survival data for TCGA-SKCM cases with and without the `chr7:g.140753336A>T` mutation \n",
    "\n",
    "- For this example we wish to use the survival analysis endpoint to compare two survival plots for TCGA-SKCM cases: one plot with cases having the `chr7:g.140753336A>T`, and the other plot for cases with out the mutation. \n",
    "- Can retrieve the `ssm_id` for a mutation from the [GDC Data Portal > Exploration](https://portal.gdc.cancer.gov/exploration) tab. \n",
    "- The API query will also print the results of a chi-squared analysis between the two subsets of cases\n",
    "    - Note that results of chi-square test are dependent on number of cases returned for each plot (`size` parameter); to choose all cases, use total number of cases in project for `size` parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://api.gdc.cancer.gov/analysis/survival?filters=%5B%7B%22op%22%3A%20%22and%22%2C%20%22content%22%3A%20%5B%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cases.project.project_id%22%2C%20%22value%22%3A%20%22TCGA-SKCM%22%7D%7D%2C%20%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22gene.ssm.ssm_id%22%2C%20%22value%22%3A%20%2284aef48f-31e6-52e4-8e05-7d5b9ab15087%22%7D%7D%5D%7D%2C%20%7B%22op%22%3A%20%22and%22%2C%20%22content%22%3A%20%5B%7B%22op%22%3A%20%22%3D%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22cases.project.project_id%22%2C%20%22value%22%3A%20%22TCGA-SKCM%22%7D%7D%2C%20%7B%22op%22%3A%20%22excludeifany%22%2C%20%22content%22%3A%20%7B%22field%22%3A%20%22gene.ssm.ssm_id%22%2C%20%22value%22%3A%20%2284aef48f-31e6-52e4-8e05-7d5b9ab15087%22%7D%7D%5D%7D%5D&fields=&size=10&format=JSON&pretty=true'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#step 1: specify and encode filters\n",
    "filters = [  \n",
    "  {  \n",
    "    \"op\":\"and\",\n",
    "    \"content\":[  \n",
    "      {  \n",
    "        \"op\":\"=\",\n",
    "        \"content\":{  \n",
    "          \"field\":\"cases.project.project_id\",\n",
    "          \"value\":\"TCGA-SKCM\"\n",
    "        }\n",
    "      },\n",
    "      {  \n",
    "        \"op\":\"=\",\n",
    "        \"content\":{  \n",
    "          \"field\":\"gene.ssm.ssm_id\",\n",
    "          \"value\":\"84aef48f-31e6-52e4-8e05-7d5b9ab15087\"\n",
    "        }\n",
    "      }\n",
    "    ]\n",
    "  },\n",
    "  {  \n",
    "    \"op\":\"and\",\n",
    "    \"content\":[  \n",
    "      {  \n",
    "        \"op\":\"=\",\n",
    "        \"content\":{  \n",
    "          \"field\":\"cases.project.project_id\",\n",
    "          \"value\":\"TCGA-SKCM\"\n",
    "        }\n",
    "      },\n",
    "      {  \n",
    "        \"op\":\"excludeifany\",\n",
    "        \"content\":{  \n",
    "          \"field\":\"gene.ssm.ssm_id\",\n",
    "          \"value\":\"84aef48f-31e6-52e4-8e05-7d5b9ab15087\"\n",
    "        }\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "]\n",
    "\n",
    "json_string=str(json.dumps(filters))\n",
    "filters_format = urllib.parse.quote(json_string.encode('utf-8'))\n",
    "\n",
    "#step 2: specify that all fields to be returned (default =  \"\") \n",
    "fields = \",\".join([\n",
    "    \"\"\n",
    "])\n",
    "\n",
    "#step 3+4: specify size=10 and format=JSON, build query url with 'analysis/survival' endpoint,\n",
    "survival_request = queryBuilder('analysis/survival', filters_format, \"\", '10', \"JSON\")\n",
    "\n",
    "#step 5: send request\n",
    "survival_result = requests.get(survival_request)\n",
    "\n",
    "#print(survival_result.text)\n",
    "survival_request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'chiSquared': 0.0026163750182500124,\n",
       " 'degreesFreedom': 1,\n",
       " 'pValue': 0.9592055847930816}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#can parse out the chi-squared test from results\n",
    "#by loading results as a JSON object and selecting\n",
    "#overallStats from results\n",
    "\n",
    "json.loads(survival_result.text)['overallStats']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='submit'>Using the GDC API to Submit Data to GDC</a>\n",
    "\n",
    "### Overview\n",
    "\n",
    "- For projects that have been approved to be included in the GDC, submitters can make use of the `submission` GDC API endpoint to submit node entities to submission projects\n",
    "- Submission will require a token downloaded from the [GDC Submission Portal](https://docs.gdc.cancer.gov/Data_Submission_Portal/Users_Guide/Data_Submission_Process/#authentication)\n",
    "- Data can be submitted in `JSON` or `TSV` format; depending on the data format, users will need to edit the `\"Content-Type\"` in the request command (see below)\n",
    "- Additionally, `JSON` and `TSV` templates for nodes to be submitted can be downloaded from the GDC Data Dictionary Viewer webpage: https://docs.gdc.cancer.gov/Data_Dictionary/viewer/#?_top=1\n",
    "- Submittable files (such as FASTQ or BAM files) should be uploaded with the [GDC Data Transfer Tool](https://gdc.cancer.gov/access-data/gdc-data-transfer-tool)\n",
    "- Additional features and more information regarding submission using the GDC API can be found here: https://docs.gdc.cancer.gov/API/Users_Guide/Submission/ \n",
    "- [Strategies for Submitting in Bulk](https://docs.gdc.cancer.gov/Data_Submission_Portal/Users_Guide/Data_Submission_Walkthrough/#strategies-for-submitting-in-bulk)\n",
    "\n",
    "### Endpoint\n",
    "\n",
    "- The format for using the GDC API Submission enpoint uses the project information, i.e. `https://api.gdc.cancer.gov/<program_name>/<project_code>`\n",
    "- For example: https://api.gdc.cancer.gov/submission/TCGA/LUAD or https://api.gdc.cancer.gov/submission/CPTAC/3 \n",
    "\n",
    "### Steps\n",
    "\n",
    "1. Read in token file\n",
    "2. Read in submission file\n",
    "3. Edit endpoint with project ID information and submit data using `POST` (JSON file submission) or `PUT` (TSV file submission) request\n",
    "\n",
    "### Example 7: Submitting a JSON Data File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#1. Read in token file\n",
    "\n",
    "token = open(\"path/to/file/gdc-user-token.txt\").read().strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#2. Read in submission file\n",
    "\n",
    "example_file_json = json.load(open(\"example_file.json\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#3. Edit endpoint and submit data using POST request\n",
    "\n",
    "ENDPT = \"https://api.gdc.cancer.gov/submission/<program_name>/<project_id>\"\n",
    "\n",
    "#submission request if data is in JSON format\n",
    "requests.post(url = ENDPT, json = example_file_json, headers={'X-Auth-Token': token, \"Content-Type\": \"application/json\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 8: Submitting a TSV Data File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#1. Read in token file\n",
    "\n",
    "token = open(\"path/to/file/gdc-user-token.txt\").read().strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#2. Read in submission file\n",
    "\n",
    "example_file_tsv = open(\"example_file.txt\", \"rb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#3. Edit endpoint and submit data using PUT request\n",
    "\n",
    "ENDPT = \"https://api.gdc.cancer.gov/submission/<program_name>/<project_id>\"\n",
    "\n",
    "#submission request if data is in TSV format\n",
    "res = requests.put(url = ENDPT, data = example_file_tsv, headers={'X-Auth-Token': token, \"Content-Type\": \"text/tsv\"})\n",
    "\n",
    "res.text"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}