Jupyter/M05-DataAnalytics/M05C-IsolationForest.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MLPqIoR78gWo"
      },
      "source": [
        "![Cloud-First](https://github.com/tulip-lab/sit742/blob/develop/Jupyter/image/CloudFirst.png?raw=1)\n",
        "\n",
        "\n",
        "# SIT742: Modern Data Science\n",
        "**(Module: Big Data Analytics)**\n",
        "\n",
        "---\n",
        "- Materials in this module include resources collected from various open-source online repositories.\n",
        "- You are free to use, change and distribute this package.\n",
        "- If you found any issue/bug for this document, please submit an issue at [tulip-lab/sit742](https://github.com/tulip-lab/sit742/issues)\n",
        "\n",
        "\n",
        "Prepared by **SIT742 Teaching Team**\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "## Session 5C: Isolation Forest"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rDh_UsLI8gWq"
      },
      "source": [
        "Let's import the required libraries first. We are importing numpy, pandas, seaborn and matplotlib. Apart form that we also need to import IsolationForest from sklearn.ensemble."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wfJIXqie8gWq"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.ensemble import IsolationForest\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yJFQIGZx8gWr"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8dgkc2V78gWr"
      },
      "source": [
        "Once the libraries are imported we need to read the data from the csv to the pandas data frame and check the first 10 rows of data.\n",
        "\n",
        "The data is a collection of Hong Kong arrival time series. This data has few anomalies (like values too high or too low) which we will be detecting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "elScHVdU8gWs"
      },
      "outputs": [],
      "source": [
        "link_to_data = 'https://raw.githubusercontent.com/tulip-lab/open-data/master/HK2012-2018/Australia.csv'\n",
        "\n",
        "# Data Preprocessing\n",
        "df = pd.read_csv(link_to_data)\n",
        "df = df[['date','arrival','Hong kong','Hong kong dollar']]\n",
        "df = df.set_index('date', drop=False)\n",
        "df.head(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1G4zj9L-8gWs"
      },
      "outputs": [],
      "source": [
        "from sklearn import preprocessing\n",
        "min_max_scaler = preprocessing.MinMaxScaler()\n",
        "x_scaled = min_max_scaler.fit_transform(df[['arrival','Hong kong','Hong kong dollar']])\n",
        "df_normal = pd.DataFrame(x_scaled,columns=['arrival','Hong kong','Hong kong dollar'])\n",
        "df_normal = df_normal.set_index(df.index.values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "85fqTpH-8gWs"
      },
      "source": [
        "## Exploratory Data Analysis\n",
        "\n",
        "To get more of an idea of the data we have plotted a violin plot of salary data as shown below. A violin plot is a method of plotting numeric data.\n",
        "\n",
        "Typically a violin plot includes all the data that is in a box plot, a marker for the median of the data, a box or marker indicating the interquartile range, and possibly all sample points, if the number of samples is not too high."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RbekE8Nc8gWt"
      },
      "outputs": [],
      "source": [
        "# we firstly try to consolidate all values from columns into one and then give a category to distingush.\n",
        "df_arr = df_normal[['arrival']]\n",
        "df_hk = df_normal[['Hong kong']]\n",
        "df_hkd = df_normal[['Hong kong dollar']]\n",
        "df_arr['category'] = 'arrival'\n",
        "df_hk['category'] = 'hongkong'\n",
        "df_hkd['category'] = 'hongkongdollar'\n",
        "df_arr.columns = ['value','category']\n",
        "df_hk.columns = ['value','category']\n",
        "df_hkd.columns = ['value','category']\n",
        "df_eda = pd.concat([df_arr,df_hk,df_hkd],axis=0)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oC8tDB0l8gWt"
      },
      "outputs": [],
      "source": [
        "ax = sns.violinplot(x=\"category\", y=\"value\", data=df_eda)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ex-J553F8gWu"
      },
      "source": [
        "To get a better idea of outliers we may like to look at a box plot as well. This is also known as box-and-whisker plot. The box in box plot shows the quartiles of the dataset, while the whiskers shows the rest of the distribution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y2MRX1un8gWu"
      },
      "outputs": [],
      "source": [
        "ax = sns.boxplot(x=\"category\", y=\"value\", data=df_eda)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJ7n2uoM8gWu"
      },
      "source": [
        "## IsolationForest\n",
        "\n",
        "Once we have completed our exploratory data analysis, it's time to define and fit the model.\n",
        "\n",
        "We'll create a model variable and instantiate the IsolationForest class. We are passing the values of four parameters to the Isolation Forest method, listed below.\n",
        "\n",
        "- Number of estimators: n_estimators refers to the number of base estimators or trees in the ensemble, i.e. the number of trees that will get built in the forest. This is an integer parameter and is optional. The default value is 100.\n",
        "\n",
        "- Max samples: max_samples is the number of samples to be drawn to train each base estimator. If max_samples is more than the number of samples provided, all samples will be used for all trees. The default value of max_samples is 'auto'. If 'auto', then max_samples=min(256, n_samples)\n",
        "\n",
        "- Contamination: This is a parameter that the algorithm is quite sensitive to; it refers to the expected proportion of outliers in the data set. This is used when fitting to define the threshold on the scores of the samples. The default value is 'auto'. If ‘auto’, the threshold value will be determined as in the original paper of Isolation Forest.\n",
        "\n",
        "- Max features: All the base estimators are not trained with all the features available in the dataset. It is the number of features to draw from the total features to train each base estimator or tree.The default value of max features is one.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6zoyV2N08gWu"
      },
      "source": [
        "In here, we will define a IsolationForest class with sklearn IsolationForest(max_samples=100)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "COzDMdoq8gWu"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import IsolationForest\n",
        "rng = np.random.RandomState(0)\n",
        "clf = IsolationForest(max_samples=100, random_state=rng)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJHPxU2h8gWv"
      },
      "source": [
        "After we defined the model above we need to train the model using the data given. For this we are using the fit() method as shown above. This method is passed one parameter, which is our data of interest (in this case, It will be the first column of the dataset -- Arrival).\n",
        "\n",
        "Once the model is trained properly it will output the IsolationForest instance as shown in the output of the cell above.\n",
        "\n",
        "### Anomaly Score\n",
        "\n",
        "Now this is the time to add the scores and anomaly column of the dataset.\n",
        "\n",
        "After the model is defined and fit, let's find the scores and anomaly column. We can find out the values of scores column by calling `decision_function()` of the trained model and passing the arrival as parameter.\n",
        "\n",
        "Similarly we can find the values of anomaly column by calling the `predict()` function of the trained model and passing the arrival as parameter.\n",
        "\n",
        "These columns are going to be added to the data frame df. After adding these two columns let's check the data frame. As expected, the data frame has three columns now: arrival, scores and anomaly. A **negative score value and a -1** for the value of anomaly columns indicate the presence of anomaly.** A value of 1 for the anomaly represents the normal data**.\n",
        "\n",
        "Each data point in the train set is assigned an anomaly score by this algorithm. We can define a threshold, and using the anomaly score, it may be possible to mark a data point as anomalous if its score is greater than the predefined threshold.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "bhHjYIzM8y6j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3FkWD8uW8gWv"
      },
      "outputs": [],
      "source": [
        "clf.fit(df[['arrival']])\n",
        "score_df = pd.DataFrame()\n",
        "score_df['scores']=clf.decision_function(df[['arrival']])\n",
        "score_df['anomaly']=clf.predict(df[['arrival']])\n",
        "score_df = score_df.set_index(df.index.values)\n",
        "score_df['arrival'] = df[['arrival']]\n",
        "score_df.head(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oNsi1EcK8gWv"
      },
      "source": [
        "### Output the Anomalies\n",
        "\n",
        "After adding the scores and anomalies for all the rows in the data, we will print the predicted anomalies. To print the predicted anomalies in the data we need to analyse the data after addition of scores and anomaly column. As you can see above for the predicted anomalies the anomaly column values would be -1 and their scores will be negative. Using this information we can print the predicted anomaly (two data points in this case) as below.\n",
        "\n",
        "Note that we could print not only the anomalous values but also their index in the dataset, which is useful information for further processing.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e4sbFM_u8gWv"
      },
      "outputs": [],
      "source": [
        "anomaly=score_df.loc[score_df['anomaly']==-1]\n",
        "anomaly_index=list(anomaly.index)\n",
        "print(anomaly)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9mBtrGQz8gWv"
      },
      "source": [
        "Let's draw the time series and also add a red dash line on the date when each amonaly happens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PjCi1pGS8gWv"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.rcParams.update({'figure.figsize':(20,5), 'figure.dpi':120})\n",
        "positions = list(range(len(score_df.index)))\n",
        "\n",
        "anomaly_index_seq = []\n",
        "for i in anomaly_index:\n",
        "    anomaly_index_seq.append(np.where((score_df.index.values == i))[0][0])\n",
        "\n",
        "series = score_df['arrival']\n",
        "ax = series.plot(rot=45)\n",
        "ax.set_xticks(positions)\n",
        "ax.set_xticklabels(score_df.index.values)\n",
        "for xc in anomaly_index_seq:\n",
        "    ax.axvline(x=xc,color='r',linestyle='dashed',linewidth=1)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxluDYVv8gWv"
      },
      "source": [
        "It could be seen that the isolation forest is able to capture the anomaly on the peak and also valleys of the arrival series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUpEbPrJ8gWw"
      },
      "source": [
        "## Another IsolationForest Example to identify the amonaly score for multi columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LnhUFvWN8gWw"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import IsolationForest\n",
        "rng = np.random.RandomState(0)\n",
        "\n",
        "# Helper function to train and predict IF model for a feature\n",
        "def train_and_predict_if(df, feature):\n",
        "    clf = IsolationForest(max_samples=100, random_state=rng)\n",
        "    clf.fit(df[[feature]])\n",
        "    pred = clf.predict(df[[feature]])\n",
        "    scores = clf.decision_function(df[[feature]])\n",
        "    stats = pd.DataFrame()\n",
        "    stats['val'] = df[feature]\n",
        "    stats['score'] = scores\n",
        "    stats['outlier'] = pred\n",
        "    stats['min'] = df[feature].min()\n",
        "    stats['max'] = df[feature].max()\n",
        "    stats['mean'] = df[feature].mean()\n",
        "    stats['feature'] = [feature] * len(df)\n",
        "    return stats\n",
        "\n",
        "# Helper function to print outliers\n",
        "def print_outliers(df, feature, n):\n",
        "    print(feature)\n",
        "    print(df[feature].head(n).to_string(), \"\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IJaQp88o8gWw"
      },
      "outputs": [],
      "source": [
        "# Run through all features and save the outlier scores for each feature\n",
        "num_columns = [i for i in list(df_normal.columns) if i not in list(df_normal.select_dtypes('object').columns) and i not in ['date']]\n",
        "result = pd.DataFrame()\n",
        "for feature in num_columns:\n",
        "    stats = train_and_predict_if(df_normal, feature)\n",
        "    result = pd.concat([result, stats])\n",
        "\n",
        "# Gather top outliers for each feature\n",
        "outliers = {team:grp.drop('feature', axis=1)\n",
        "       for team, grp in result.sort_values(by='score').groupby('feature')}\n",
        "\n",
        "# Print the top 10 outlier samples for a few selected features\n",
        "n_outliers = 10\n",
        "print_outliers(outliers, \"arrival\", n_outliers)\n",
        "print_outliers(outliers, \"Hong kong\", n_outliers)\n",
        "print_outliers(outliers, \"Hong kong dollar\", n_outliers)"
      ]
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}