Jupyter/M05-DataAnalytics/M05A-TimeSeries.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2TE_jkg_Bqx"
      },
      "source": [
        "![Cloud-First](../image/CloudFirst.png) \n",
        "\n",
        "\n",
        "# SIT742: Modern Data Science \n",
        "**(Module: Big Data Analytics)**\n",
        "\n",
        "---\n",
        "- Materials in this module include resources collected from various open-source online repositories.\n",
        "- You are free to use, change and distribute this package.\n",
        "- If you found any issue/bug for this document, please submit an issue at [tulip-lab/sit742](https://github.com/tulip-lab/sit742/issues)\n",
        "\n",
        "\n",
        "Prepared by **SIT742 Teaching Team**\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Session 5A: Time Series\n",
        "\n",
        "**The purpose of this session is to illustrate**\n",
        "\n",
        "1. Understand the Basic Time series Structure."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YXm62ZJz_JYZ"
      },
      "source": [
        "##What is Time Series?\n",
        "\n",
        "\n",
        "**Time series data, which means any information collected over a regular interval of time, in their operations.**\n",
        "\n",
        "Before we are jumping into the time series forecasting, it is essential for us to understand the time series structure, feature and also the ways of processing. We will start on some simple but interesting exploration tasks first.\n",
        "\n",
        "Tasks List:\n",
        "\n",
        "> Reading and Displaying Data\n",
        "\n",
        "> Stationarity\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eftW2PdLSteQ"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wv2PyXvQCFfU"
      },
      "source": [
        "##**Task1: Reading and Displaying Data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Lr6kRqYE_Yx"
      },
      "source": [
        "####To start, let’s import the Pandas library and read the airline passenger data into a data frame:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R6_YPrNkE-up"
      },
      "outputs": [],
      "source": [
        "import pandas as pd \n",
        "df = pd.read_csv(\"https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/timeseries-data.txt\")\n",
        "print(df.head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zb6tk0b2GhJX"
      },
      "source": [
        "We can see that the data contains a column labeled “Date” that contains datetime. In that column, the dates are formatted as year–month-day. We also see that the data starts in the year 1959.\n",
        "\n",
        "The second column is labeled “Births,” and it contains the number of new births for the year–month-day as quantitative column. \n",
        "\n",
        "From the above data, we could see **two important** findings:\n",
        "\n",
        "\n",
        "1.   Datetime (Time Stamps) is the key index to sort the time sereis data, and also the important feature for us to obtain more informative patterns.\n",
        "2.   Quantitative Unit Variable(s) is the column to provide the quantitative information on each timestamp.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hfkg3LuGIuXf"
      },
      "source": [
        "###1.1 Timestamp Formatting\n",
        "\n",
        "As we see above, the \"Date\" column has the formate of year-month-day, it will allow us to aggregate the \"Births\" with a grain level of \"day\". In some situation, mutiple grain level you might want to try, therefore,\n",
        "the formatting on timestamps is important step to process the time series data.\n",
        "\n",
        "We will do below subtasks and plot the time series accordingly:\n",
        "\n",
        "\n",
        "\n",
        "*   Format \"Date\" to month level\n",
        "*   Aggregate the \"Births\" with month level\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xvBfiPvbE2XR"
      },
      "outputs": [],
      "source": [
        "# we first format the Date column to a datetime format which pandas could read as \"datetime\"\n",
        "df['Date'] = pd.to_datetime(df['Date'], format='%Y-%m-%d')\n",
        "print(df.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "anT1ty94NVqD"
      },
      "outputs": [],
      "source": [
        "# we extract the year from Date (make sure datetime format for Date)\n",
        "df['year_month'] = df['Date'].dt.to_period('M')\n",
        "print(df.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sAi7X89UNv7c"
      },
      "outputs": [],
      "source": [
        "# Let's aggregate the Births in year level\n",
        "df.groupby(['year_month'])['Births'].agg('sum')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j5d8ZsTMPyjK"
      },
      "outputs": [],
      "source": [
        "#Let's try to plot the births in month level, to avoid change the interval on df, we create a new dataframe for this task\n",
        "df_year_month = pd.DataFrame(df.groupby(['year_month'])['Births'].agg('sum'),columns=['Births'])\n",
        "\n",
        "#import the matplotlib seaborn for plotting \n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns \n",
        "plt.rcParams[\"figure.figsize\"] = (20,3)\n",
        "#we might need to convert the index of the new dataframe to timestamp or datetime\n",
        "sns.lineplot(x=df_year_month.index.to_timestamp(), y=df_year_month.Births)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HQ3-NTAXRPzb"
      },
      "outputs": [],
      "source": [
        "#let's compare with the original time series with day level\n",
        "df.index = df['Date']\n",
        "sns.lineplot(x=df.index, y=df.Births)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tyNQreQSUHiv"
      },
      "source": [
        "**Summary:**\n",
        "\n",
        "Comparing the original time series plot with the month level one, \n",
        "it is clear to see that the month level time series could be easy to find the pattern -- Q3 - Q4 is the peak for new births.\n",
        "The above finding inspired us to think: \n",
        "***How to find the trend pattern from a given time series? what is the seasonality***?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4cimcNEVHWN"
      },
      "source": [
        "##**Task 2 Stationary**\n",
        "\n",
        "Stationarity is a key part of time series analysis. Simply put, stationarity means that the manner in which time series data changes is constant. A stationary time series will not have any trends or seasonal patterns. You should check for stationarity because it not only makes modeling time series easier, but it is an underlying assumption in many time series methods. Specifically, stationarity is assumed for a wide variety of time series forecasting methods including autoregressive moving average (ARMA).\n",
        "\n",
        "For obtaining the stationary timeseries data, we firstly learn some \n",
        "basic terms regarding the feature of the time series:\n",
        "\n",
        "\n",
        "\n",
        "1.   Seasonality\n",
        "2.   Trend\n",
        "3.   Random Noise\n",
        "2.   Rolling Statistics\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pA-L-rrfWC11"
      },
      "source": [
        "###2.1 Seasonality, Trend and Random noise\n",
        "\n",
        "Seasonality is a simple term that means while predicting a time series data there are some months in a particular domain where the output value is at a peak as compared to other months.\n",
        "For example, you might always see that the Dec and Nov is the peak for retail shoppings every year.\n",
        "\n",
        "The trend is also one of the important factors which describes that there is certainly increasing or decreasing trend time series. The trend could describe\n",
        "the pattern of either increasing or decreasing from your time series clearly.\n",
        "\n",
        "Random noise is what the time series left when both trend and seasonality\n",
        "has been removed. It is a series that’s not predictable.\n",
        "\n",
        "To be able to generate the seasonality and trend from the time series, \n",
        "we usually need to do the **decomposition** first.\n",
        "The time series could be seen as either **Addictive and Multiplicative** time series. \n",
        "\n",
        "Addictive: Y = Trend + Seasonality + Random noise, \n",
        "Multiplicative: Y = Trend * Seasonality * Random noise\n",
        "\n",
        "To do: \n",
        "\n",
        "\n",
        "\n",
        "1.   Decomposite the birth time series with addictive mode\n",
        "2.   Decomposite the birth time series with multiplicative mode\n",
        "3.   Generate the seasonality plot\n",
        "4.   Generate the trend plot\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MJcEVDGzcKMK"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from statsmodels.tsa.seasonal import seasonal_decompose\n",
        "from dateutil.parser import parse\n",
        "\n",
        "# let's use the birth data with year-month-day format in the decomposition\n",
        "df = df[['Births']]\n",
        "# Additive Decomposition\n",
        "add_result = seasonal_decompose(df['Births'], model='additive')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NePgnJCocvGh"
      },
      "outputs": [],
      "source": [
        "plt.rcParams[\"figure.figsize\"] = (15,10)\n",
        "add_result.plot().suptitle('Births Decompose', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YHmTrJQycqnA"
      },
      "outputs": [],
      "source": [
        "# Multiplicative Decomposition \n",
        "mul_result = seasonal_decompose(df['Births'], model='multiplicative')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qmA3rssoeCZp"
      },
      "outputs": [],
      "source": [
        "mul_result.plot().suptitle('Births Decompose', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oqSe3ylNX3Ck"
      },
      "source": [
        "**Summary:**\n",
        "\n",
        "From the above two decompositions, it could be seen that the seasonality of birth data rotates on every 5 days. The trend shows that the births on period of Sep - Oct are significantly higher. \n",
        "Next, we will see wether the trend is more stationary than the original time series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FODVACtyguNu"
      },
      "source": [
        "###Task 2.1 Stationary Test\n",
        "\n",
        "Stationary is constantly mean and constant variance. Adfuller is a simple test which tells that if the time series is stationary which is a kind of hypothesis testing. The Null hypothesis is time series are non-stationary. If the p-value is less than 5 percent then reject the NULL hypothesis else accept the NULL hypothesis\n",
        "\n",
        "We will compare the trend with original time series, \n",
        "also we will see whether a rolling average could also offer the stationary time series.\n",
        "\n",
        "To do:\n",
        "\n",
        "\n",
        "1.   Generate rolling average time series \n",
        "2.   Run adfuller test on trend, original and rolling average time series\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iQvEiLnmiI91"
      },
      "outputs": [],
      "source": [
        "# we could use the pandas rolling mean to obtain the moving \n",
        "rolling_mean = df.rolling(7).mean()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TrjAgu4Fjlg1"
      },
      "outputs": [],
      "source": [
        "# Let's plot the rolling mean, trend and origianl time series in one plot\n",
        "\n",
        "plt.plot(df, color=\"blue\",label=\"Original birth data\")\n",
        "plt.plot(rolling_mean, color=\"red\", label=\"Rolling Mean birth data\")\n",
        "plt.plot(mul_result.trend, color=\"green\", label=\"Rolling Mean birth data\")\n",
        "plt.legend(loc=\"best\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sd2GRPyCkNjS"
      },
      "outputs": [],
      "source": [
        "from statsmodels.tsa.stattools import adfuller\n",
        "#let's pass the time series into the adfuller test function, firstly let's test the original\n",
        "adft = adfuller(df.Births,autolag=\"AIC\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KWl4l8xOnBY8"
      },
      "outputs": [],
      "source": [
        "# let's check the test results\n",
        "\n",
        "output_df = pd.DataFrame({\"Values\":[adft[0],adft[1],adft[2],adft[3], adft[4]['1%'], adft[4]['5%'], adft[4]['10%']]  , \"Metric\":[\"Test Statistics\",\"p-value\",\"No. of lags used\",\"Number of observations used\", \n",
        "                                                        \"critical value (1%)\", \"critical value (5%)\", \"critical value (10%)\"]})\n",
        "print(output_df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3F-GODq9nKVZ"
      },
      "outputs": [],
      "source": [
        "# let's test with trend and rolling mean\n",
        "adft_trend = adfuller(mul_result.trend.values[~np.isnan(mul_result.trend.values)],autolag=\"AIC\")\n",
        "adft_rolling = adfuller(rolling_mean.Births.values[~np.isnan(rolling_mean.Births.values)],autolag=\"AIC\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vO6QTQb-nb0g"
      },
      "outputs": [],
      "source": [
        "output_df_trend = pd.DataFrame({\"Values\":[adft_trend[0],adft_trend[1],adft_trend[2],adft_trend[3], adft_trend[4]['1%'], adft_trend[4]['5%'], adft_trend[4]['10%']]  , \"Metric\":[\"Test Statistics\",\"p-value\",\"No. of lags used\",\"Number of observations used\", \n",
        "                                                        \"critical value (1%)\", \"critical value (5%)\", \"critical value (10%)\"]})\n",
        "print(output_df_trend)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8tXElFYDoERA"
      },
      "outputs": [],
      "source": [
        "output_df_rolling = pd.DataFrame({\"Values\":[adft_rolling[0],adft_rolling[1],adft_rolling[2],adft_rolling[3], adft_rolling[4]['1%'], adft_rolling[4]['5%'], adft_rolling[4]['10%']]  , \"Metric\":[\"Test Statistics\",\"p-value\",\"No. of lags used\",\"Number of observations used\", \n",
        "                                                        \"critical value (1%)\", \"critical value (5%)\", \"critical value (10%)\"]})\n",
        "print(output_df_rolling)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BKE39GaonLUq"
      },
      "source": [
        "We can see that our data is not stationary from the fact that our p-value is greater than 5 percent and the test statistic is greater than the critical value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ct8zhD1GpoEa"
      },
      "source": [
        "**Summary:**\n",
        "\n",
        "From the adfuller test, we could see that the original time series has t-test value smaller than the 1% and 5%, also P value is extremely small. Therefore,\n",
        "the original time series is more stationary than trend and rolling average. Why is it? Let's check the definition of stationary -- value changes is constant, so it means the variance is close to 0. Trend and rolling average's changes via datetime is not a constant.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}