Exploratory_data_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TOvht7vqQGdR"
   },
   "source": [
    "# Exploratory data analysis in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dB_j6LtTTO5j"
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8hg00soETQ3z"
   },
   "source": [
    "**What is Exploratory Data Analysis ?**\n",
    "\n",
    "Exploratory Data Analysis or (EDA) is how we as data analysts gain an understanding of the data sets we are working with by summarizing their main characteristics, grooming the data and often plotting them visually. This step is very important especially when we start to develop the modeling in order to apply Machine learning. Plotting in EDA consists of Histograms, Box plot, Scatter plot and many more. It often takes much time to explore the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZfelutoyTS25"
   },
   "source": [
    "**How to perform Exploratory Data Analysis ?**\n",
    "\n",
    " Well, the answer is it depends on the data set that you are working. There is no one method or common methods in order to perform EDA, whereas in this tutorial you can understand some common methods and plots that would be used in the EDA process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n3VfNkBBw15s"
   },
   "source": [
    "**What data are we exploring today ?**\n",
    "\n",
    "\n",
    "\n",
    "This data set contains more of 10,000 rows and more than 10 columns which contains features of the car such as Engine Fuel Type, Engine HP, Transmission Type, highway MPG, city MPG and many more. So in this tutorial, we will explore the data and make it ready for modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CQDO4JCqTThV"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FPSqz1wzTXvz"
   },
   "source": [
    "## 1. Importing the required libraries for EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9eLMx1Ebwa92"
   },
   "source": [
    "Below are the libraries that are used in order to perform EDA (Exploratory data analysis) in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "id": "GGyDovL2QDLa",
    "outputId": "6732d162-14f7-432a-cc64-e45b7d5d9f47"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns                       #visualisation\n",
    "import matplotlib.pyplot as plt             #visualisation\n",
    "#%matplotlib inline     \n",
    "sns.set(color_codes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ye6eXuohTd5Q"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Y3Z2DbKTfJt"
   },
   "source": [
    "## 2. Loading the data into the data frame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ko5zGJFCySaz"
   },
   "source": [
    "Loading the data into the pandas data frame is certainly one of the most important steps in EDA, as we can see that the value from the data set is comma-separated. So all we have to do is to just read the CSV into a data frame and pandas data frame does the job for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LgzUzD61IM8h"
   },
   "source": [
    "To get or load the dataset into the notebook, all I did was one trivial step. In Google Colab at the left-hand side of the notebook, you will find a > (greater than symbol). When you click that you will find a tab with three options, you just have to select Files. Then you can easily upload your file with the help of the Upload option. No need to mount to the google drive or use any specific libraries just upload the data set and your job is done. One thing to remember in this step is that uploaded files will get deleted when this runtime is recycled. This is how I got the data set into the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 428
    },
    "id": "0oVZnezwQ159",
    "outputId": "ce37fd4a-4b62-487a-f9cd-55d5ede5090a"
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"eda_data.csv\")\n",
    "# To display the top 5 rows \n",
    "df.head(5)               "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 413
    },
    "id": "Fm-9dzdTRKpe",
    "outputId": "7892eaf7-0605-4b92-e139-cf0553041e51"
   },
   "outputs": [],
   "source": [
    "df.tail(5)                        # To display the botton 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JjQnr4SPzaL5"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vAmC369yTpMF"
   },
   "source": [
    "## 3. Checking the types of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9ESKxikIzA1d"
   },
   "source": [
    "Here we check for the datatypes because sometimes the MSRP or the price of the car would be stored as a string, if in that case, we have to convert that string to the integer data only then we can plot the data via a graph. Here, in this case, the data is already in integer format so nothing to worry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 312
    },
    "id": "qPbKQ0noRptD",
    "outputId": "6a5aea47-ad0c-4118-8471-d91b6432b339"
   },
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "beRJyQAezdX8"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QoHuBhXxT5E9"
   },
   "source": [
    "## 4. Dropping irrelevant columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_3cy877Mze4H"
   },
   "source": [
    "This step is certainly needed in every EDA because sometimes there would be many columns that we never use in such cases dropping is the only solution. In this case, the columns such as Engine Fuel Type, Market Category, Vehicle style, Popularity, Number of doors, Vehicle Size doesn't make any sense to me so I just dropped for this instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "uvSkK8swTr9H",
    "outputId": "1734f538-a4a1-45b9-d656-7ddc0124dc35"
   },
   "outputs": [],
   "source": [
    "df = df.drop(['Engine Fuel Type', 'Market Category', 'Vehicle Style', 'Popularity', 'Number of Doors', 'Vehicle Size'], axis=1)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "20OeQBpWz89v"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "caAownWdUZso"
   },
   "source": [
    "## 5. Renaming the columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UABLiEcyz-2G"
   },
   "source": [
    "In this instance, most of the column names are very confusing to read, so I just tweaked their column names. This is a good approach it improves the readability of the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "1wHW41whURub",
    "outputId": "dec9b1b3-e344-4b33-92fd-6e9f4c03c878"
   },
   "outputs": [],
   "source": [
    "df = df.rename(columns={\"Engine HP\": \"HP\", \"Engine Cylinders\": \"Cylinders\", \"Transmission Type\": \"Transmission\", \"Driven_Wheels\": \"Drive Mode\",\"highway MPG\": \"MPG-H\", \"city mpg\": \"MPG-C\", \"MSRP\": \"Price\" })\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5tcGiOmV0afN"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uiW7x_O4WIDX"
   },
   "source": [
    "## 6. Dropping the duplicate rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9LpR5NW70hXm"
   },
   "source": [
    "This is often a handy thing to do because a huge data set as in this case contains more than 10, 000 rows often have some duplicate data which might be disturbing, so here I remove all the duplicate value from the data-set. For example prior to removing I had 11914 rows of data but after removing the duplicates 10925 data meaning that I had 989 of duplicate data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "1nBN_BCDWSmv",
    "outputId": "9a070a7d-a4d4-45c7-cac8-acb2c4db0e72"
   },
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "yB8t6o0wH7If",
    "outputId": "d4778cd1-5372-4e27-db62-20635493786e"
   },
   "outputs": [],
   "source": [
    "duplicate_rows_df = df[df.duplicated()]\n",
    "print(\"number of duplicate rows: \", duplicate_rows_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "chhNvMFCIzqI"
   },
   "source": [
    "Now let us remove the duplicate data because it's ok to remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "kuobmetTV820",
    "outputId": "c6d6ce5c-6a38-4cd2-ee99-151124a1f84d"
   },
   "outputs": [],
   "source": [
    "df.count()      # Used to count the number of rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_MJKjbzHI40K"
   },
   "source": [
    "So seen above there are 11914 rows and we are removing 989 rows of duplicate data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "OiOsEF6WVTSj",
    "outputId": "a527c1a2-5d74-42bb-99e2-8112ebffb871"
   },
   "outputs": [],
   "source": [
    "df = df.drop_duplicates()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "2gMM4lb0Vzor",
    "outputId": "f04e1803-e7de-4cbf-fdeb-8449a8051a07"
   },
   "outputs": [],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zCLUdZOQ1PDP"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JkXUQtyQW3Dy"
   },
   "source": [
    "## 7. Dropping the missing or null values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K5pKvJi41YCp"
   },
   "source": [
    "This is mostly similar to the previous step but in here all the missing values are detected and are dropped later. Now, this is not a good approach to do so, because many people just replace the missing values with the mean or the average of that column, but in this case, I just dropped that missing values. This is because there is nearly 100 missing value compared to 10, 000 values this is a small number and this is negligible so I just dropped those values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "Tzdlg-1OWjMz",
    "outputId": "7375ab6e-1473-4346-e5b5-0c61189cc716"
   },
   "outputs": [],
   "source": [
    "print(df.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iWJqTVxTJQnO"
   },
   "source": [
    "This is the reason in the above step while counting both Cylinders and Horsepower (HP) had 10856 and 10895 over 10925 rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "KbbV0xHPWoad",
    "outputId": "17dda8ec-1282-4814-de79-8f5e1aff3a5f"
   },
   "outputs": [],
   "source": [
    "df = df.dropna()    # Dropping the missing values.\n",
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2t7L9l2mJSoX"
   },
   "source": [
    "Now we have removed all the rows which contain the Null or N/A values (Cylinders and Horsepower (HP))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "V-DmX1O4Wtox",
    "outputId": "2d50fc20-3535-413b-e317-75a7f94fa2a2"
   },
   "outputs": [],
   "source": [
    "print(df.isnull().sum())   # After dropping the values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bk8RAHqQJVJK"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Py3sQc_ZxyU"
   },
   "source": [
    "## 8. Detecting Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1g6CJPofZzHN"
   },
   "source": [
    "An outlier is a point or set of points that are different from other points. Sometimes they can be very high or very low. It's often a good idea to detect and remove the outliers. Because outliers are one of the primary reasons for resulting in a less accurate model. Hence it's a good idea to remove them. The outlier detection and removing that I am going to perform is called IQR score technique. Often outliers can be seen with visualizations using a box plot. Shown below are the box plot of MSRP, Cylinders, Horsepower and EngineSize. Herein all the plots, you can find some points are outside the box they are none other than outliers. The technique of finding and removing outlier that I am performing in this assignment is taken help of a tutorial from[ towards data science](https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 698
    },
    "id": "vtxX_y6zZ2ri",
    "outputId": "b204a305-b669-4e0a-80d8-3b3c3bd06cde"
   },
   "outputs": [],
   "source": [
    "sns.boxplot(x=df['Price'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "id": "9HtvQYVHZ8u5",
    "outputId": "3dc30a01-6fb1-41d9-dec8-0ceeead6a358"
   },
   "outputs": [],
   "source": [
    "sns.boxplot(x=df['HP'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "id": "Xz3MwIjbaBUr",
    "outputId": "335d1e55-55a3-4e61-8401-a414580d9d62"
   },
   "outputs": [],
   "source": [
    "sns.boxplot(x=df['Cylinders'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139
    },
    "id": "cFG9hck7aHUx",
    "outputId": "7c09ff98-3725-4d56-9b97-ef6b43d0e17b"
   },
   "outputs": [],
   "source": [
    "Q1 = df.quantile(0.25)\n",
    "Q3 = df.quantile(0.75)\n",
    "IQR = Q3 - Q1\n",
    "print(IQR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kCv110_cJiDz"
   },
   "source": [
    "Don't worry about the above values because it's not important to know each and every one of them because it's just important to know how to use this technique in order to remove the outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "igh_mRXeaJrI",
    "outputId": "44660b71-2eb7-4387-c599-e527640a7c21"
   },
   "outputs": [],
   "source": [
    "df = df[~((df < (Q1 - 1.5 * IQR)) |(df > (Q3 + 1.5 * IQR))).any(axis=1)]\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Nh93DCGJkqT"
   },
   "source": [
    "As seen above there were around 1600 rows were outliers. But you cannot completely remove the outliers because even after you use the above technique there maybe 1–2 outlier unremoved but that ok because there were more than 100 outliers. Something is better than nothing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z87kHbgvaQbb"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WqFPRda8eEp_"
   },
   "source": [
    "## 9. Plot different features against one another (scatter), against frequency (histogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z-W6Q9-hJosZ"
   },
   "source": [
    "### Histogram\n",
    "\n",
    "Histogram refers to the frequency of occurrence of variables in an interval. In this case, there are mainly 10 different types of car manufacturing companies, but it is often important to know who has the most number of cars. To do this histogram is one of the trivial solutions which lets us know the total number of car manufactured by a different company."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 431
    },
    "id": "dAnd4DSyeHDb",
    "outputId": "44b04e39-9dc7-40fc-9ddb-b7182f4f6e1f"
   },
   "outputs": [],
   "source": [
    "df.Make.value_counts().nlargest(40).plot(kind='bar', figsize=(10,5))\n",
    "plt.title(\"Number of cars by make\")\n",
    "plt.ylabel('Number of cars')\n",
    "plt.xlabel('Make');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c37WtYYWJuAQ"
   },
   "source": [
    "### Heat Maps\n",
    "\n",
    "Heat Maps is a type of plot which is necessary when we need to find the dependent variables. One of the best way to find the relationship between the features can be done using heat maps. In the below heat map we know that the price feature depends mainly on the Engine Size, Horsepower, and Cylinders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "id": "yhHfLVTj6nmy",
    "outputId": "50a65ae0-841a-42ec-87e6-1a99da1ea57b"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "c= df.corr()\n",
    "sns.heatmap(c,cmap=\"BrBG\",annot=True)\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ROThOLQfRZw"
   },
   "source": [
    "### Scatterplot\n",
    "\n",
    "We generally use scatter plots to find the correlation between two variables. Here the scatter plots are plotted between Horsepower and Price and we can see the plot below. With the plot given below, we can easily draw a trend line. These features provide a good scattering of points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 401
    },
    "id": "2aOfHNFefSrX",
    "outputId": "4b5286f0-5419-48bc-d2de-476aeb36f022"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,6))\n",
    "ax.scatter(df['HP'], df['Price'])\n",
    "ax.set_xlabel('HP')\n",
    "ax.set_ylabel('Price')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hQXy8o_gKFS5"
   },
   "source": [
    "**Hence the above are some of the steps involved in Exploratory data analysis, these are some general steps that you must follow in order to perform EDA. There are many more yet to come but for now, this is more than enough idea as to how to perform a good EDA given any data sets. Stay tuned for more updates.**\n",
    "\n",
    "## Thank you."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Exploratory_data_Analysis.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}