Final_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build an ML Pipeline for Airfoil Noise Prediction\n",
    "\n",
    "In this project, we use a modified version of the NASA Airfoil Self-Noise dataset to create a model that will predict the SoundLevel based on other columns in the dataset. After training the model, we assess its performance using relevant metrics to gauge accuracy and effectiveness. The model is saved for future use, ensuring it can be retrieved and deployed in real-world applications to make predictions on new data.\n",
    "\n",
    "This project has four parts: \n",
    "\n",
    "- Part 1 - Perform ETL activity\n",
    "  - Load a csv dataset\n",
    "  - Remove duplicates if any\n",
    "  - Drop rows with null values if any\n",
    "  - Make transformations\n",
    "  - Store the cleaned data in parquet format\n",
    "- Part 2 - Create a Machine Learning Pipeline\n",
    "  - Create a machine learning pipeline for prediction\n",
    "- Part 3 - Evaluate the Model\n",
    "  - Evaluate the model using relevant metrics\n",
    "- Part 4 - Persist the Model\n",
    "  - Save the model for future production use\n",
    "  - Load and verify the stored model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preliminaries: Installing libraries and downloading data\n",
    "\n",
    "Install the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install pyspark==3.1.2 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the required data file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'NASA_airfoil_noise_raw.csv'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download the `NASA_airfoil_noise_raw.csv` file\n",
    "import wget\n",
    "wget.download(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-BD0231EN-Coursera/datasets/NASA_airfoil_noise_raw.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries\n",
    "\n",
    "Importing the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import findspark\n",
    "import warnings\n",
    "\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "\n",
    "# Suppress generated warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "findspark.init()\n",
    "\n",
    "# import functions/Classes for sparkml\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.feature import StandardScaler\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "\n",
    "# import functions/Classes for pipeline creation\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.pipeline import PipelineModel\n",
    "\n",
    "# import functions/Classes for metrics\n",
    "from pyspark.ml.evaluation import RegressionEvaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a spark session\n",
    "\n",
    "Ignore any warnings by SparkSession command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/04/26 02:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .appName(\"Airfoil Noise Prediction\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 1 - Perform ETL activity\n",
    "\n",
    "Our initial step involves data cleaning, where we eliminate duplicate rows and those with missing values. This process ensures that the data remains reliable and consistent for subsequent analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a csv dataset\n",
    "\n",
    "* Using the `spark.read.csv` function we load the data into a dataframe\n",
    "* The `header=True` indicates that there is a header row in our csv file\n",
    "* The `inferSchema=True` tells spark to automatically determine the data types of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "df = spark.read.csv(\"NASA_airfoil_noise_raw.csv\", header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show top 5 rows from the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+-------------+-----------+------------------+-----------------------+----------+\n",
      "|Frequency|AngleOfAttack|ChordLength|FreeStreamVelocity|SuctionSideDisplacement|SoundLevel|\n",
      "+---------+-------------+-----------+------------------+-----------------------+----------+\n",
      "|      800|          0.0|     0.3048|              71.3|             0.00266337|   126.201|\n",
      "|     1000|          0.0|     0.3048|              71.3|             0.00266337|   125.201|\n",
      "|     1250|          0.0|     0.3048|              71.3|             0.00266337|   125.951|\n",
      "|     1600|          0.0|     0.3048|              71.3|             0.00266337|   127.591|\n",
      "|     2000|          0.0|     0.3048|              71.3|             0.00266337|   127.461|\n",
      "+---------+-------------+-----------+------------------+-----------------------+----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show total number of rows in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1522\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "rowcount1 = df.count()\n",
    "print(rowcount1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove duplicates, if any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = df.dropDuplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show total number of rows in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 6:===================================================>   (186 + 7) / 200]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1503\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "rowcount2 = df.count()\n",
    "print(rowcount2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop rows with null values, if any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show total number of rows in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 9:====================================================>  (191 + 9) / 200]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1499\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "rowcount3 = df.count()\n",
    "print(rowcount3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make transformations\n",
    "\n",
    "* Rename the column `SoundLevel` to `SoundLevelDecibels`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = df.withColumnRenamed(\"SoundLevel\", \"SoundLevelDecibels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the cleaned data in parquet format\n",
    "\n",
    "* Save the dataframe as `NASA_airfoil_noise_cleaned.parquet`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "df.write.parquet(\"NASA_airfoil_noise_cleaned.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 1 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 1 - Evaluation\n",
      "Total rows =  1522\n",
      "Total rows after dropping duplicate rows =  1503\n",
      "Total rows after dropping duplicate rows and rows with null values =  1499\n",
      "New column name =  SoundLevelDecibels\n",
      "NASA_airfoil_noise_cleaned.parquet exists : True\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 1 - Evaluation\")\n",
    "\n",
    "print(\"Total rows = \", rowcount1)\n",
    "print(\"Total rows after dropping duplicate rows = \", rowcount2)\n",
    "print(\"Total rows after dropping duplicate rows and rows with null values = \", rowcount3)\n",
    "print(\"New column name = \", df.columns[-1])\n",
    "\n",
    "print(\"NASA_airfoil_noise_cleaned.parquet exists :\", os.path.isdir(\"NASA_airfoil_noise_cleaned.parquet\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 2 - Create a  Machine Learning Pipeline\n",
    "\n",
    "Following this, we’ll create a Machine Learning pipeline comprising three stages, one of which involves regression. This pipeline will facilitate the development of a model that predicts SoundLevel based on other columns in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a machine learning pipeline for prediction\n",
    "\n",
    "First, load data from \"NASA_airfoil_noise_cleaned.parquet\" into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = spark.read.parquet(\"NASA_airfoil_noise_cleaned.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show total number of rows in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 14:===========================================>              (6 + 2) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1499\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "rowcount4 = df.count()\n",
    "print(rowcount4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the VectorAssembler pipeline stage\n",
    "\n",
    "Stage 1:\n",
    "* Assemble the input columns into a single column `features`\n",
    "* Use all the columns except `SoundLevelDecibels` as input features\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "    inputCols=[\n",
    "        \"Frequency\",\n",
    "        \"AngleOfAttack\",\n",
    "        \"ChordLength\",\n",
    "        \"FreeStreamVelocity\",\n",
    "        \"SuctionSideDisplacement\"\n",
    "    ],\n",
    "    outputCol=\"features\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the StandardScaler pipeline stage\n",
    "\n",
    "Stage 2:\n",
    "* Scale the `features` using standard scaler and store in `scaledFeatures` column\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the StandardScaler pipeline stage\n",
    "\n",
    "Stage 3:\n",
    "* Create a LinearRegression stage to predict `SoundLevelDecibels` using `scaledFeatures`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "lr = LinearRegression(featuresCol=\"scaledFeatures\", labelCol=\"SoundLevelDecibels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a pipeline using the above three stages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(stages=[assembler, scaler, lr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data\n",
    "* Split the data into training and testing sets with 70:30 split\n",
    "* Set the value of seed to 42\n",
    "\n",
    "DO NOT set the value of seed to any other value other than 42."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(trainingData, testingData) = df.randomSplit([0.7, 0.3], seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the pipeline using the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/04/26 02:41:57 WARN util.Instrumentation: [e2f62d28] regParam is zero, which might cause numerical instability and overfitting.\n",
      "[Stage 19:>                                                         (0 + 8) / 8]24/04/26 02:42:07 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "24/04/26 02:42:07 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "24/04/26 02:42:09 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "24/04/26 02:42:09 WARN netlib.LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "pipelineModel = pipeline.fit(trainingData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 2 - Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 2 - Evaluation\n",
      "Total rows =  1499\n",
      "Pipeline Stage 1 =  VectorAssembler\n",
      "Pipeline Stage 2 =  StandardScaler\n",
      "Pipeline Stage 3 =  LinearRegression\n",
      "Label column =  SoundLevelDecibels\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 2 - Evaluation\")\n",
    "print(\"Total rows = \", rowcount4)\n",
    "ps = [str(x).split(\"_\")[0] for x in pipeline.getStages()]\n",
    "\n",
    "print(\"Pipeline Stage 1 = \", ps[0])\n",
    "print(\"Pipeline Stage 2 = \", ps[1])\n",
    "print(\"Pipeline Stage 3 = \", ps[2])\n",
    "\n",
    "print(\"Label column = \", lr.getLabelCol())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 3 - Evaluate the Model\n",
    "\n",
    "After training the model, we will assess its accuracy and effectiveness using suitable metrics. Subsequently, we’ll save the model for future use, ensuring that it can be stored and retrieved later. This allows us to deploy the trained model in real-world applications and make predictions on new data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the model using relevant metrics\n",
    "\n",
    "Make predictions on testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictions = pipelineModel.transform(testingData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the Mean Square Error (MSE)\n",
    "\n",
    "* Lower the value the better the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 26:==================================================>       (7 + 1) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE = 22.593754071348812\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"SoundLevelDecibels\",\n",
    "    predictionCol=\"prediction\",\n",
    "    metricName=\"mse\"\n",
    ")\n",
    "\n",
    "mse = evaluator.evaluate(predictions)\n",
    "print(f\"MSE = {mse}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the Mean Absolute Error (MAE)\n",
    "\n",
    "* Lower the value the better the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 28:==================================================>       (7 + 1) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 3.7336902294631287\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"SoundLevelDecibels\",\n",
    "    predictionCol=\"prediction\",\n",
    "    metricName=\"mae\"\n",
    ")\n",
    "\n",
    "mae = evaluator.evaluate(predictions)\n",
    "print(f\"MAE = {mae}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the R-Squared (R2)\n",
    "\n",
    "* Higher values indicate better performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 30:===========================================>              (6 + 2) / 8]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R Squared = 0.5426016508689058\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"SoundLevelDecibels\",\n",
    "    predictionCol=\"prediction\",\n",
    "    metricName=\"r2\"\n",
    ")\n",
    "\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "print(f\"R Squared = {r2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 3 - Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 3 - Evaluation\n",
      "Mean Squared Error =  22.59\n",
      "Mean Absolute Error =  3.73\n",
      "R Squared =  0.54\n",
      "Intercept =  132.6\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 3 - Evaluation\")\n",
    "\n",
    "print(\"Mean Squared Error = \", round(mse,2))\n",
    "print(\"Mean Absolute Error = \", round(mae,2))\n",
    "print(\"R Squared = \", round(r2,2))\n",
    "\n",
    "lrModel = pipelineModel.stages[-1]\n",
    "\n",
    "print(\"Intercept = \", round(lrModel.intercept,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 4 - Persist the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the model for future production use\n",
    "\n",
    "* Save the pipeline model as \"Final_Project\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# create folder to save model\n",
    "!mkdir -p Final_Project\n",
    "\n",
    "# Persist the model to the path \"./Final_Project/\"\n",
    "pipelineModel.write().overwrite().save(\"./Final_Project/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load and verify the stored model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "loadedPipelineModel = PipelineModel.load(\"./Final_Project/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the loaded pipeline model and make predictions using testingData\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictions = loadedPipelineModel.transform(testingData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the predictions\n",
    "\n",
    "* Show top 5 rows from the predections dataframe\n",
    "* Display only the label column and predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 52:>                                                         (0 + 1) / 1]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+\n",
      "|SoundLevelDecibels|        prediction|\n",
      "+------------------+------------------+\n",
      "|           127.315|123.64344009624753|\n",
      "|           119.975|123.48695788614877|\n",
      "|           121.783|124.38983849684254|\n",
      "|           127.224|121.44706993294302|\n",
      "|           122.229|125.68312652454188|\n",
      "+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "predictions.select(\"SoundLevelDecibels\",\"prediction\").show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 4 - Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Part 4 - Evaluation\n",
      "Number of stages in the pipeline =  3\n",
      "Coefficient for Frequency is -3.9728\n",
      "Coefficient for AngleOfAttack is -2.4775\n",
      "Coefficient for ChordLength is -3.3818\n",
      "Coefficient for FreeStreamVelocity is 1.5789\n",
      "Coefficient for SuctionSideDisplacement is -1.6465\n"
     ]
    }
   ],
   "source": [
    "print(\"Part 4 - Evaluation\")\n",
    "\n",
    "loadedmodel = loadedPipelineModel.stages[-1]\n",
    "totalstages = len(loadedPipelineModel.stages)\n",
    "inputcolumns = loadedPipelineModel.stages[0].getInputCols()\n",
    "\n",
    "print(\"Number of stages in the pipeline = \", totalstages)\n",
    "for i,j in zip(inputcolumns, loadedmodel.coefficients):\n",
    "    print(f\"Coefficient for {i} is {round(j,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Spark Session\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change Log\n",
    "\n",
    "\n",
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2024-04-24  | 0.2  | Pravin Regismond | Modified to fulfill project requirements |\n",
    "| 2023-05-26  | 0.1  | Ramesh Sannareddy | Initial Version Created |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2023 IBM Corporation. All rights reserved."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}