09-getting-started-with-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Visualization with Modern Data Science\n",
    "\n",
    "> Getting Started with Pandas\n",
    "\n",
    "Yao-Jen Kuo <yaojenkuo@ntu.edu.tw> from [DATAINPOINT](https://www.datainpoint.com/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About `pandas`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What is `pandas`?\n",
    "\n",
    "> Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more.\n",
    "\n",
    "Source: <https://github.com/pandas-dev/pandas>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Why `pandas`?\n",
    "\n",
    "Python used to have a weak spot in its analysis capability due to it did not have an appropriate structure handling the common tabular datasets. Pythonists had to switch to a more data-centric language like R or Matlab during the analysis stage until the presence of `pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Import Pandas with `import` command\n",
    "\n",
    "Pandas is officially aliased as `pd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## If Pandas is not installed, we will encounter a `ModuleNotFoundError`\n",
    "\n",
    "```\n",
    "Traceback (most recent call last):\n",
    "  File \"<stdin>\", line 1, in <module>\n",
    "ModuleNotFoundError: No module named 'pandas'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Use `pip install` at Terminal to install pandas\n",
    "\n",
    "```bash\n",
    "pip install pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Check version and its installation file path\n",
    "\n",
    "- `__version__` attribute\n",
    "- `__file__` attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.3\n",
      "/Users/kuoyaojen/miniconda3/lib/python3.11/site-packages/pandas/__init__.py\n"
     ]
    }
   ],
   "source": [
    "print(pd.__version__)\n",
    "print(pd.__file__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What does `pandas` mean?\n",
    "\n",
    "![](https://media.giphy.com/media/46Zj6ze2Z2t4k/giphy.gif)\n",
    "\n",
    "Source: <https://giphy.com/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Turns out its naming has nothing to do with panda the animal, it refers to three primary class customed by its author [Wes McKinney](https://wesmckinney.com/)\n",
    "\n",
    "- **Pan**el(Deprecated since version 0.20.0)\n",
    "- **Da**taFrame\n",
    "- **S**eries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## In order to master `pandas`, it is vital to understand the relationships between `Index`, `ndarray`, `Series`, and `DataFrame`\n",
    "\n",
    "- An `Index` and a `ndarray` assembles a `Series`.\n",
    "- A couple of `Series` that sharing the same `Index` can then form a `DataFrame`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `Index` from Pandas\n",
    "\n",
    "The simplest way to create an `Index` is using `pd.Index()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.indexes.base.Index"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prime_indices = pd.Index([2, 3, 5, 7, 11, 13, 17, 19, 23, 29])\n",
    "type(prime_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## An `Index` is like a combination of `tuple` and `set`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index does not support mutable operations\n"
     ]
    }
   ],
   "source": [
    "# immutable\n",
    "prime_indices = pd.Index([2, 3, 5, 7, 11, 13, 17, 19, 23, 29])\n",
    "try:\n",
    "    prime_indices[-1] = 31\n",
    "except TypeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index([3, 5, 7, 11, 13, 17, 19, 23, 29], dtype='int64')\n",
      "Index([1, 2, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29], dtype='int64')\n",
      "Index([1, 2, 9, 15, 21, 25, 27], dtype='int64')\n",
      "Index([2], dtype='int64')\n",
      "Index([1, 9, 15, 21, 25, 27], dtype='int64')\n"
     ]
    }
   ],
   "source": [
    "# Index has the characteristics of a set\n",
    "odd_indices = pd.Index(range(1, 30, 2))\n",
    "print(prime_indices.intersection(odd_indices))         # prime_indices & odd_indices\n",
    "print(prime_indices.union(odd_indices))                # prime_indices | odd_indices\n",
    "print(prime_indices.symmetric_difference(odd_indices)) # prime_indices ^ odd_indices\n",
    "print(prime_indices.difference(odd_indices))\n",
    "print(odd_indices.difference(prime_indices))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `Series` from Pandas\n",
    "\n",
    "The simplest way to create a `Series` is using `pd.Series()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "0     2\n",
      "1     3\n",
      "2     5\n",
      "3     7\n",
      "4    11\n",
      "5    13\n",
      "6    17\n",
      "7    19\n",
      "8    23\n",
      "9    29\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "prime_series = pd.Series([2, 3, 5, 7, 11, 13, 17, 19, 23, 29])\n",
    "print(type(prime_series))\n",
    "print(prime_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## A `Series` is a combination of `Index` and `ndarray`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.indexes.range.RangeIndex'>\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print(type(prime_series.index))\n",
    "print(type(prime_series.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## The index of a `Series` can be customized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1      2\n",
       "2      3\n",
       "3      5\n",
       "4      7\n",
       "5     11\n",
       "6     13\n",
       "7     17\n",
       "8     19\n",
       "9     23\n",
       "10    29\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prime_series = pd.Series([2, 3, 5, 7, 11, 13, 17, 19, 23, 29],\n",
    "                        index=range(1, 11))\n",
    "prime_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1      2\n",
       "2      3\n",
       "3      5\n",
       "4      7\n",
       "5     11\n",
       "6     13\n",
       "7     17\n",
       "8     19\n",
       "9     23\n",
       "10    29\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prime_series = pd.Series([2, 3, 5, 7, 11, 13, 17, 19, 23, 29])\n",
    "prime_series.index = range(1, 11)\n",
    "prime_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Indexing a `Series`\n",
    "\n",
    "Indexing via index positions or index labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n",
      "11\n"
     ]
    }
   ],
   "source": [
    "prime_series = pd.Series([2, 3, 5, 7, 11],\n",
    "                        index=[\"1st\", \"2nd\", \"3rd\", \"4th\", \"5th\"])\n",
    "print(prime_series[-1])\n",
    "print(prime_series[\"5th\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Slicing a `Series`\n",
    "\n",
    "Slicing via index positions excludes `stop`, while slicing via index labels includes `stop`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st    2\n",
      "2nd    3\n",
      "dtype: int64\n",
      "1st    2\n",
      "2nd    3\n",
      "3rd    5\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(prime_series[:2])\n",
    "print(prime_series[\"1st\":\"3rd\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## A `Series` contains a `ndarray` therefore they can be manipulated in the same way \n",
    "\n",
    "- Vectorization and broadcasting.\n",
    "- Fancy indexing.\n",
    "- Boolean indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1st      4\n",
       "2nd      9\n",
       "3rd     25\n",
       "4th     49\n",
       "5th    121\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Vectorization and broadcasting\n",
    "prime_series**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st     2\n",
      "2nd     3\n",
      "5th    11\n",
      "dtype: int64\n",
      "1st     2\n",
      "2nd     3\n",
      "5th    11\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Fancy indexing\n",
    "print(prime_series[[0, 1, 4]])\n",
    "print(prime_series[[\"1st\", \"2nd\", \"5th\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2nd     3\n",
       "3rd     5\n",
       "4th     7\n",
       "5th    11\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Boolean indexing\n",
    "prime_series[prime_series % 2 == 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `DataFrame` from Pandas\n",
    "\n",
    "The simplest way to create a `DataFrame` is using `pd.DataFrame()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# column-wise\n",
    "movie_df = pd.DataFrame()\n",
    "movie_df[\"title\"] = [\"The Shawshank Redemption\", \"The Dark Knight\", \"Schindler's List\", \"Forrest Gump\", \"Inception\"]\n",
    "movie_df[\"imdb_rating\"] = [9.3, 9.0, 8.9, 8.8, 8.7]\n",
    "type(movie_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>imdb_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Inception</td>\n",
       "      <td>8.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      title  imdb_rating\n",
       "0  The Shawshank Redemption          9.3\n",
       "1           The Dark Knight          9.0\n",
       "2          Schindler's List          8.9\n",
       "3              Forrest Gump          8.8\n",
       "4                 Inception          8.7"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `DataFrame` from Pandas\n",
    "\n",
    "Creating a `DataFrame` with a `dict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# column-wise\n",
    "movie_dict = {\n",
    "    \"title\": [\"The Shawshank Redemption\", \"The Dark Knight\", \"Schindler's List\", \"Forrest Gump\", \"Inception\"],\n",
    "    \"imdb_rating\": [9.3, 9.0, 8.9, 8.8, 8.7]\n",
    "}\n",
    "movie_df = pd.DataFrame(movie_dict)\n",
    "type(movie_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>imdb_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Inception</td>\n",
       "      <td>8.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      title  imdb_rating\n",
       "0  The Shawshank Redemption          9.3\n",
       "1           The Dark Knight          9.0\n",
       "2          Schindler's List          8.9\n",
       "3              Forrest Gump          8.8\n",
       "4                 Inception          8.7"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `DataFrame` from Pandas\n",
    "\n",
    "Creating a `DataFrame` with a `list` of dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# row-wise\n",
    "movie_list = [\n",
    "    {\"title\": \"The Shawshank Redemption\", \"imdb_rating\": 9.3},\n",
    "    {\"title\": \"The Dark Knight\", \"imdb_rating\": 9.0},\n",
    "    {\"title\": \"Schindler's List\", \"imdb_rating\": 8.9},\n",
    "    {\"title\": \"Forrest Gump\", \"imdb_rating\": 8.8},\n",
    "    {\"title\": \"Inception\", \"imdb_rating\": 8.7}\n",
    "]\n",
    "movie_df = pd.DataFrame(movie_list)\n",
    "type(movie_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>imdb_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Inception</td>\n",
       "      <td>8.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      title  imdb_rating\n",
       "0  The Shawshank Redemption          9.3\n",
       "1           The Dark Knight          9.0\n",
       "2          Schindler's List          8.9\n",
       "3              Forrest Gump          8.8\n",
       "4                 Inception          8.7"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## A `DataFrame` is a combination of multiple `Series` sharing the same `Index`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.indexes.range.RangeIndex'>\n",
      "<class 'pandas.core.series.Series'>\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "print(type(movie_df.index))\n",
    "print(type(movie_df[\"title\"]))\n",
    "print(type(movie_df[\"imdb_rating\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Review of the definition of modern data science\n",
    "\n",
    "> Modern data science is a huge field, it invovles applications and tools like importing, tidying, transformation, visualization, modeling, and communication. Surrounding all these is programming.\n",
    "\n",
    "![Imgur](https://i.imgur.com/din6Ig6.png)\n",
    "\n",
    "Source: [R for Data Science](https://r4ds.had.co.nz/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Key functionalities analysts rely on `pandas` are\n",
    "\n",
    "- Importing\n",
    "- Tidying\n",
    "- Transforming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Tidying and transforming together is also known as WRANGLING\n",
    "\n",
    "![](https://media.giphy.com/media/MnlZWRFHR4xruE4N2Z/giphy.gif)\n",
    "\n",
    "Source: <https://giphy.com>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Importing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `pandas` has massive functions importing tabular data\n",
    "\n",
    "- Flat text file\n",
    "- Database table\n",
    "- Spreadsheet\n",
    "- JSON\n",
    "- HTML `<table></table>` tags\n",
    "- ...etc.\n",
    "\n",
    "Source: <https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using `read_csv` function for flat text files\n",
    "\n",
    "```python\n",
    "csv_file_path = \"PATH/TO/YOUR/CSVs/FILE\"\n",
    "pd.read_csv(csv_file_path)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "(53385, 7)\n"
     ]
    }
   ],
   "source": [
    "csv_file_path = \"data/presidents.csv\"\n",
    "presidents = pd.read_csv(csv_file_path)\n",
    "print(type(presidents))\n",
    "print(presidents.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using `read_sql` function for database tables\n",
    "\n",
    "```python\n",
    "import sqlite3\n",
    "\n",
    "conn = sqlite3.connect('YOUR_DATABASE.db')\n",
    "sql_query = \"\"\"\n",
    "SELECT * \n",
    "  FROM YOUR_TABLE\n",
    " LIMIT 10;\n",
    "\"\"\"\n",
    "pd.read_sql(sql_query, conn)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "(5, 2)\n"
     ]
    }
   ],
   "source": [
    "import sqlite3\n",
    "\n",
    "conn = sqlite3.connect(\"data/taiwan_election_2024.db\")\n",
    "sql_query = \"\"\"\n",
    "SELECT *\n",
    "  FROM election_types;\n",
    "\"\"\"\n",
    "election_types = pd.read_sql(sql_query, conn)\n",
    "print(type(election_types))\n",
    "print(election_types.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using `read_excel` function for spreadsheets\n",
    "\n",
    "```python\n",
    "excel_file_path = \"PATH/TO/YOUR/EXCEL/FILE\"\n",
    "pd.read_excel(excel_file_path)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kuoyaojen/miniconda3/lib/python3.11/site-packages/openpyxl/styles/stylesheet.py:226: UserWarning: Workbook contains no default style, apply openpyxl's default\n",
      "  warn(\"Workbook contains no default style, apply openpyxl's default\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "(1781, 14)\n"
     ]
    }
   ],
   "source": [
    "presidents_excel = pd.read_excel(\"data/總統-A05-4-候選人得票數一覽表-各投開票所(臺北市).xlsx\")\n",
    "print(type(presidents_excel))\n",
    "print(presidents_excel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using `read_json` function for JSON\n",
    "\n",
    "```python\n",
    "json_file_path = \"PATH/TO/YOUR/JSON/FILE\"\n",
    "pd.read_json(json_file_path)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using `read_html` function for HTML `<table></table>` tags\n",
    "\n",
    "> The `<table>` tag defines an HTML table. An HTML table consists of one `<table>` element and one or more `<tr>`, `<th>`, and `<td>` elements. The `<tr>` element defines a table row, the `<th>` element defines a table header, and the `<td>` element defines a table cell.\n",
    "\n",
    "Source: <https://www.w3schools.com/default.asp>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}