Final Assignment Webscraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/Images/SN_logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\">\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Extracting Stock Data Using a Web Scraping</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all stock data is available via API in this assignment; you will use web-scraping to obtain financial data. You will be quizzed on your results.  \n",
    " Using beautiful soup we will extract historical share data from a web-page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    \n",
    "1. Extracting data using Beautiful soup\n",
    "   \n",
    "    <ul> \n",
    "    <li> Downloading the Webpage Using Requests Library </li>\n",
    "    <li> Parsing Webpage HTML Using BeautifulSoup </li>\n",
    "    <li> Extracting Data and Building DataFrame </li>\n",
    "\n",
    "    </ul>\n",
    "\n",
    "2. Extracting data using pandas\n",
    "3. Exercise\n",
    "<p>\n",
    "    Estimated Time Needed: <strong>30 min</strong></p>\n",
    "</div>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (1.4.2) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['bs4==4.10.0']\n",
      "\n",
      "\u001b[?25l\u001b[2K\u001b[0G[+] 0.0s\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[0G[+] 0.1s\n",
      "pkgs/main/linux-64 \u001b[33m━━━━━━━━━━━╸\u001b[0m\u001b[90m━━━━━━━━━━━━━\u001b[0m   0.0 B /  ??.?MB @  ??.?MB/s  0.1s\n",
      "pkgs/main/noarch   \u001b[33m━━━━━━━━━╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━\u001b[0m   0.0 B /  ??.?MB @  ??.?MB/s  0.1s\n",
      "pkgs/r/linux-64    \u001b[90m━━━━━━━━━━━━━━╸\u001b[0m\u001b[33m━━━━━━━━━━\u001b[0m   0.0 B /  ??.?MB @  ??.?MB/s  0.1s\n",
      "pkgs/r/noarch      \u001b[90m━━━━━━╸\u001b[0m\u001b[33m━━━━━━━━━━━━━━━╸\u001b[0m\u001b[90m━━\u001b[0m   0.0 B /  ??.?MB @  ??.?MB/s  0.1s\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[0Gpkgs/main/linux-64                                            No change\n",
      "pkgs/r/linux-64                                               No change\n",
      "pkgs/r/noarch                                                 No change\n",
      "pkgs/main/noarch                                              No change\n",
      "\u001b[?25h\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  All requested packages already installed\n",
      "\n",
      "\u001b[?25l\u001b[2K\u001b[0G\u001b[?25h\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (1.4.2) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['html5lib==1.1']\n",
      "\n",
      "pkgs/main/linux-64                                          Using cache\n",
      "pkgs/main/noarch                                            Using cache\n",
      "pkgs/r/linux-64                                             Using cache\n",
      "pkgs/r/noarch                                               Using cache\n",
      "\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  All requested packages already installed\n",
      "\n",
      "\u001b[?25l\u001b[2K\u001b[0G\u001b[?25hRequirement already satisfied: lxml==4.6.4 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (4.6.4)\n"
     ]
    }
   ],
   "source": [
    "#!pip install pandas==1.3.3\n",
    "#!pip install requests==2.26.0\n",
    "!mamba install bs4==4.10.0 -y\n",
    "!mamba install html5lib==1.1 -y \n",
    "!pip install lxml==4.6.4\n",
    "#!pip install plotly==5.3.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will extract Netflix stock data [https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> \n",
    "    \n",
    "#### In this example, we are using yahoo finance website and looking to extract Netflix data.\n",
    "\n",
    "</center>\n",
    "    <br>\n",
    "\n",
    "  <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/Images/netflix.png\"> </center> \n",
    "  \n",
    "<center> Fig:- Table that we need to extract </center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the following webpage we have a table with columns name (Date, Open, High, Low, close, adj close volume) out of which we must extract following columns  \n",
    "\n",
    "* Date \n",
    "\n",
    "* Open  \n",
    "\n",
    "* High \n",
    "\n",
    "* Low \n",
    "\n",
    "* Close \n",
    "\n",
    "* Volume \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Steps to be followed for extracting data\n",
    "1. Send an HTTP request to the webpage using the requests library.\n",
    "2. Parse the HTML content of the webpage using BeautifulSoup.\n",
    "3. Identify the HTML tags that contain the data you want to extract.\n",
    "4. Use BeautifulSoup methods to extract the data from the HTML tags.\n",
    "5. Print the extracted data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step-1 Send an HTTP Request to the webpage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using Request library for sending an HTTP request to the webpage.<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The requests.get() method takes a URL as its first argument, which specifies the location of the resource to be retrieved. In this case, the value of the url variable is passed as the argument to the requests.get() method, as we've stored a webpage URL in a url variable.\n",
    "\n",
    "we have used .text method for extracting the HTML content as a string in order to make it readable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data  = requests.get(url).text\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step:-2 Parse the HTML content\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<hr>\n",
    "<center>\n",
    "\n",
    "# What is parsing?\n",
    "In simple words, parsing refers to the process of analyzing a string of text or a data structure, usually following a set of rules or grammar, to understand its structure and meaning.\n",
    "Parsing involves breaking down a piece of text or data into its individual components or elements, and then analyzing those components to extract the desired information or to understand their relationships and meanings.</center>\n",
    "<hr>\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will take the raw HTML content of a webpage or a string of HTML code which needs to be parsed and transformed into a structured, hierarchical format that can be more easily analyzed and manipulated in Python. This can be done using a Python library called <b>Beautiful Soup</b>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to parse the data using Beautiful soup library?\n",
    "* Create a new Beautiful soup object.\n",
    "<br>\n",
    "<br>\n",
    "<b>Note:- </b>To create a Beautiful Soup object in Python, you need to pass two arguments to its constructor:\n",
    "\n",
    "1. The HTML or XML content that you want to parse as a string.\n",
    "2. The name of the parser that you want to use to parse the HTML or XML content. This argument is optional, and if you don't specify a parser, Beautiful Soup will use the default HTML parser included with the library.\n",
    "here in this lab we are using \"html5lib\" parser.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step-3 Identify the HTML tags\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated above webpage consist of table so, we will be scrapping the content of the HTML webpage and convert the table into a dataframe.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will creates an empty DataFrame using the <b> pd.DataFrame() </b> function with the following columns.\n",
    "* \"Date\"\n",
    "* \"Open\"\n",
    "* \"High\" \n",
    "* \"Low\" \n",
    "* \"Close\"\n",
    "* \"Volume\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "netflix_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<hr>\n",
    "<center>\n",
    "\n",
    "### Working on HTML table  </center>\n",
    "<br>\n",
    "\n",
    "These are the following tags which are used while creating HTML tables.\n",
    "\n",
    "* &lt;table&gt; tag: This tag is root tag used to define the start and end of the table. All the content of the table is enclosed within these tags. \n",
    "\n",
    "\n",
    "* &lt;tr&gt; tag: This tag is used to define a table row. Each row of the table is defined within this tag.\n",
    "\n",
    "* &lt;td&gt; tag: This tag is used to define a table cell. Each cell of the table is defined within this tag. You can specify the content of the cell between the opening and closing <td> tags.\n",
    "\n",
    "* &lt;th&gt; tag: This tag is used to define a header cell in the table. The header cell is used to describe the contents of a column or row. By default, the text inside a <th> tag is bold and centered.\n",
    "\n",
    "* &lt;tbody&gt; tag: This is the main content of the table, which is defined using the <tbody> tag. It contains one or more rows of <tr> elements.\n",
    "\n",
    "<hr>\n",
    "<hr>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step-4 Use Beautiful soup method for extracting data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We will use <b>find()</b> and <b>find_all()</b> methods of the BeautifulSoup object to locate the table body and table row respectively in the HTML. \n",
    "   * The <i>find() method </i> will return particular tag content.\n",
    "   * The <i>find_all()</i> method returns a list of all matching tags in the HTML.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# First we isolate the body of the table which contains all the information\n",
    "# Then we loop through each row and find all the column values for each row\n",
    "for row in soup.find(\"tbody\").find_all('tr'):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    # Finally we append the data of each row to the table\n",
    "    netflix_data = netflix_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step-5 Print the Extracted Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the DataFrame using head() or tail() function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78,560,600</td>\n",
       "      <td>528.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66,927,600</td>\n",
       "      <td>502.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111,573,300</td>\n",
       "      <td>513.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90,183,900</td>\n",
       "      <td>521.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61,902,300</td>\n",
       "      <td>538.85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low   Close       Volume Adj Close\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21   78,560,600    528.21\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81   66,927,600    502.81\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47  111,573,300    513.47\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66   90,183,900    521.66\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85   61,902,300    538.85"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting data using `pandas` library\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the pandas `read_html` function from pandas library and use the URL for extracting data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "\n",
    "## What is read_html in pandas library?\n",
    "`pd.read_html(url)` is a function provided by the pandas library in Python that is used to extract tables from HTML web pages. It takes in a URL as input and returns a list of all the tables found on the webpage. \n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can convert the BeautifulSoup object to a string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(str(soup))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there is only one table on the page, we just take the first table in the list returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close*</th>\n",
       "      <th>Adj Close**</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78560600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66927600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111573300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90183900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61902300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low  Close* Adj Close**     Volume\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21      528.21   78560600\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81      502.81   66927600\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47      513.47  111573300\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66      521.66   90183900\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85      538.85   61902300"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_dataframe = read_html_pandas_data[0]\n",
    "\n",
    "netflix_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `requests` library to download the webpage [https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html). Save the text of the response as a variable named `html_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url= \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html\"\n",
    "html_data = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the html data using `beautiful_soup`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(html_data, 'html5lib')\n",
    "#print (soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 1</b> What is the content of the title attribute?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Amazon.com, Inc. (AMZN) Stock Historical Prices & Data - Yahoo Finance'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using beautiful soup extract the table with historical share prices and store it into a dataframe named `amazon_data`. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. Fill in each variable with the correct data from the list `col`. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "for row in soup.find(\"tbody\").find_all(\"tr\"):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    amazon_data = amazon_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first five rows of the `amazon_data` dataframe you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jan 01, 2021</td>\n",
       "      <td>3,270.00</td>\n",
       "      <td>3,363.89</td>\n",
       "      <td>3,086.00</td>\n",
       "      <td>3,206.20</td>\n",
       "      <td>71,528,900</td>\n",
       "      <td>3,206.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dec 01, 2020</td>\n",
       "      <td>3,188.50</td>\n",
       "      <td>3,350.65</td>\n",
       "      <td>3,072.82</td>\n",
       "      <td>3,256.93</td>\n",
       "      <td>77,556,200</td>\n",
       "      <td>3,256.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nov 01, 2020</td>\n",
       "      <td>3,061.74</td>\n",
       "      <td>3,366.80</td>\n",
       "      <td>2,950.12</td>\n",
       "      <td>3,168.04</td>\n",
       "      <td>90,810,500</td>\n",
       "      <td>3,168.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Oct 01, 2020</td>\n",
       "      <td>3,208.00</td>\n",
       "      <td>3,496.24</td>\n",
       "      <td>3,019.00</td>\n",
       "      <td>3,036.15</td>\n",
       "      <td>116,226,100</td>\n",
       "      <td>3,036.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sep 01, 2020</td>\n",
       "      <td>3,489.58</td>\n",
       "      <td>3,552.25</td>\n",
       "      <td>2,871.00</td>\n",
       "      <td>3,148.73</td>\n",
       "      <td>115,899,300</td>\n",
       "      <td>3,148.73</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date      Open      High       Low     Close       Volume Adj Close\n",
       "0  Jan 01, 2021  3,270.00  3,363.89  3,086.00  3,206.20   71,528,900  3,206.20\n",
       "1  Dec 01, 2020  3,188.50  3,350.65  3,072.82  3,256.93   77,556,200  3,256.93\n",
       "2  Nov 01, 2020  3,061.74  3,366.80  2,950.12  3,168.04   90,810,500  3,168.04\n",
       "3  Oct 01, 2020  3,208.00  3,496.24  3,019.00  3,036.15  116,226,100  3,036.15\n",
       "4  Sep 01, 2020  3,489.58  3,552.25  2,871.00  3,148.73  115,899,300  3,148.73"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 2</b> What is the name of the columns of the dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(amazon_data.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 3</b> What is the `Open` of the last row of the amazon_data dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data[\"Open\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork900-2023-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "Azim Hirjani<br>\n",
    "Akansha yadav\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By    |       Change Description              |\n",
    "| ----------------- | ------- | ------------- | ------------------------------------- |\n",
    "|  02-05-2023       |   1.3   | Akansha yadav | Updated Lab content under maintenance |\n",
    "|  2021-06-09       | 1.2     | Lakshmi Holla |   Added URL in question 3             |\n",
    "|  2020-11-10       | 1.1     | Malika Singla |   Deleted the Optional part           |\n",
    "|  2020-08-27       | 1.0     | Malika Singla |   Added lab to GitLab                 |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n",
    "\n",
    "<p>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}