lectures/mehrdad/23_deployment-conclusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](../img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 23: Deployment and conclusion\n",
    "\n",
    "UBC 2024 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports, LOs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:38.995420Z",
     "iopub.status.busy": "2024-04-11T20:08:38.994926Z",
     "iopub.status.idle": "2024-04-11T20:08:40.277131Z",
     "shell.execute_reply": "2024-04-11T20:08:40.275929Z",
     "shell.execute_reply.started": "2024-04-11T20:08:38.995373Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "from sklearn.pipeline import Pipeline, make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning objectives\n",
    "\n",
    "- Describe the **goals** and **challenges** of model deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Try out this moment predictor\n",
    "\n",
    "https://class-demo-330.onrender.com\n",
    "\n",
    "- In this lecture, we will show how to set up/develop this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### What is deployment?\n",
    "\n",
    "- After we train a model, we want to use it!\n",
    "- The user likely does not want to install your Python stack, train your model.\n",
    "- You don't necessarily want to share your dataset.\n",
    "- So we need to do two things:\n",
    "  1. **Save/store** your model for later use.\n",
    "  2. Make the saved model conveniently **accessible**.\n",
    "\n",
    "We will use [Joblib](https://joblib.readthedocs.io/) for (1) and [Flask](https://flask.palletsprojects.com/) & [render](https://render.com/) for (2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Requirements (I already did these)\n",
    "\n",
    "- render account. Register [here](https://render.com/).\n",
    "- More python installations:\n",
    "\n",
    "`conda install -n cpsc330 -c anaconda joblib flask flask-wtf`\n",
    "\n",
    "<!-- pip install Flask -->\n",
    "<!-- pip install Flask-WTF -->\n",
    "<!-- pip install joblib -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Demo: Deploying moment classification model  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Building a model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Recall the multi-class classification problem using the [HappyDB](https://www.kaggle.com/ritresearch/happydb) corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:40.279489Z",
     "iopub.status.busy": "2024-04-11T20:08:40.279137Z",
     "iopub.status.idle": "2024-04-11T20:08:40.778058Z",
     "shell.execute_reply": "2024-04-11T20:08:40.776742Z",
     "shell.execute_reply.started": "2024-04-11T20:08:40.279455Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wid</th>\n",
       "      <th>reflection_period</th>\n",
       "      <th>original_hm</th>\n",
       "      <th>moment</th>\n",
       "      <th>modified</th>\n",
       "      <th>num_sentence</th>\n",
       "      <th>target</th>\n",
       "      <th>predicted_category</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27676</th>\n",
       "      <td>206</td>\n",
       "      <td>24h</td>\n",
       "      <td>We had a serious talk with some friends of our...</td>\n",
       "      <td>We had a serious talk with some friends of our...</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "      <td>bonding</td>\n",
       "      <td>bonding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27678</th>\n",
       "      <td>45</td>\n",
       "      <td>24h</td>\n",
       "      <td>I meditated last night.</td>\n",
       "      <td>I meditated last night.</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>leisure</td>\n",
       "      <td>leisure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27697</th>\n",
       "      <td>498</td>\n",
       "      <td>24h</td>\n",
       "      <td>My grandmother start to walk from the bed afte...</td>\n",
       "      <td>My grandmother start to walk from the bed afte...</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>affection</td>\n",
       "      <td>affection</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27705</th>\n",
       "      <td>5732</td>\n",
       "      <td>24h</td>\n",
       "      <td>I picked my daughter up from the airport and w...</td>\n",
       "      <td>I picked my daughter up from the airport and w...</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>bonding</td>\n",
       "      <td>affection</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27715</th>\n",
       "      <td>2272</td>\n",
       "      <td>24h</td>\n",
       "      <td>when i received flowers from my best friend</td>\n",
       "      <td>when i received flowers from my best friend</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>bonding</td>\n",
       "      <td>bonding</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        wid reflection_period  \\\n",
       "hmid                            \n",
       "27676   206               24h   \n",
       "27678    45               24h   \n",
       "27697   498               24h   \n",
       "27705  5732               24h   \n",
       "27715  2272               24h   \n",
       "\n",
       "                                             original_hm  \\\n",
       "hmid                                                       \n",
       "27676  We had a serious talk with some friends of our...   \n",
       "27678                            I meditated last night.   \n",
       "27697  My grandmother start to walk from the bed afte...   \n",
       "27705  I picked my daughter up from the airport and w...   \n",
       "27715        when i received flowers from my best friend   \n",
       "\n",
       "                                                  moment  modified  \\\n",
       "hmid                                                                 \n",
       "27676  We had a serious talk with some friends of our...      True   \n",
       "27678                            I meditated last night.      True   \n",
       "27697  My grandmother start to walk from the bed afte...      True   \n",
       "27705  I picked my daughter up from the airport and w...      True   \n",
       "27715        when i received flowers from my best friend      True   \n",
       "\n",
       "       num_sentence     target predicted_category  \n",
       "hmid                                               \n",
       "27676             2    bonding            bonding  \n",
       "27678             1    leisure            leisure  \n",
       "27697             1  affection          affection  \n",
       "27705             1    bonding          affection  \n",
       "27715             1    bonding            bonding  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"../data/cleaned_hm.csv\", index_col=0)\n",
    "sample_df = df.dropna()\n",
    "sample_df.head()\n",
    "sample_df = sample_df.rename(\n",
    "    columns={\"cleaned_hm\": \"moment\", \"ground_truth_category\": \"target\"}\n",
    ")\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:40.779902Z",
     "iopub.status.busy": "2024-04-11T20:08:40.779590Z",
     "iopub.status.idle": "2024-04-11T20:08:40.790099Z",
     "shell.execute_reply": "2024-04-11T20:08:40.788705Z",
     "shell.execute_reply.started": "2024-04-11T20:08:40.779880Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "target\n",
       "affection           4810\n",
       "achievement         4276\n",
       "bonding             1750\n",
       "enjoy_the_moment    1514\n",
       "leisure             1306\n",
       "nature               252\n",
       "exercise             217\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_df[\"target\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a multiclass classification problem! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:40.792255Z",
     "iopub.status.busy": "2024-04-11T20:08:40.791611Z",
     "iopub.status.idle": "2024-04-11T20:08:40.805940Z",
     "shell.execute_reply": "2024-04-11T20:08:40.803930Z",
     "shell.execute_reply.started": "2024-04-11T20:08:40.792231Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "train_df, test_df = train_test_split(sample_df, test_size=0.2, random_state=123)\n",
    "X_train_happy, y_train_happy = train_df[\"moment\"], train_df[\"target\"]\n",
    "X_test_happy, y_test_happy = test_df[\"moment\"], test_df[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:40.808883Z",
     "iopub.status.busy": "2024-04-11T20:08:40.808103Z",
     "iopub.status.idle": "2024-04-11T20:08:40.815662Z",
     "shell.execute_reply": "2024-04-11T20:08:40.814266Z",
     "shell.execute_reply.started": "2024-04-11T20:08:40.808821Z"
    }
   },
   "outputs": [],
   "source": [
    "pipe_lr = make_pipeline(\n",
    "    CountVectorizer(stop_words=\"english\"),\n",
    "    LogisticRegression(max_iter=2000),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:40.817643Z",
     "iopub.status.busy": "2024-04-11T20:08:40.817124Z",
     "iopub.status.idle": "2024-04-11T20:08:44.791587Z",
     "shell.execute_reply": "2024-04-11T20:08:44.789777Z",
     "shell.execute_reply.started": "2024-04-11T20:08:40.817599Z"
    }
   },
   "outputs": [],
   "source": [
    "pipe_lr.fit(X_train_happy, y_train_happy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:44.808751Z",
     "iopub.status.busy": "2024-04-11T20:08:44.804048Z",
     "iopub.status.idle": "2024-04-11T20:08:45.035332Z",
     "shell.execute_reply": "2024-04-11T20:08:45.034348Z",
     "shell.execute_reply.started": "2024-04-11T20:08:44.808707Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9528318584070796"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr.score(X_train_happy, y_train_happy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.036708Z",
     "iopub.status.busy": "2024-04-11T20:08:45.036284Z",
     "iopub.status.idle": "2024-04-11T20:08:45.088909Z",
     "shell.execute_reply": "2024-04-11T20:08:45.087384Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.036680Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8169911504424778"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr.score(X_test_happy, y_test_happy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training on the full corpus\n",
    "\n",
    "- Ideally, you'll try all different models, fine tune the most promising models and deploy the best performing model.  \n",
    "- Sometimes before deploying a model people train it on the full dataset. \n",
    "    - This is probably a good idea, because more data is better.\n",
    "    - It's also a little scary, because we can't test this new model.\n",
    "- Here I'm just deploying the model trained on the training set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Saving the model \n",
    "\n",
    "- If we want to deploy a model, we need to save it. \n",
    "- We are using `joblib` for that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.092690Z",
     "iopub.status.busy": "2024-04-11T20:08:45.090942Z",
     "iopub.status.idle": "2024-04-11T20:08:45.208314Z",
     "shell.execute_reply": "2024-04-11T20:08:45.206587Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.092639Z"
    }
   },
   "outputs": [],
   "source": [
    "import joblib\n",
    "\n",
    "with open(\"web_api/moment_predictor.joblib\", \"wb\") as f:\n",
    "    joblib.dump(pipe_lr, f)\n",
    "with open(\"web_application/moment_predictor.joblib\", \"wb\") as f:\n",
    "    joblib.dump(pipe_lr, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define a function that accepts input data as a dictionary and returns a prediction:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading our saved model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function to get predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.210645Z",
     "iopub.status.busy": "2024-04-11T20:08:45.210316Z",
     "iopub.status.idle": "2024-04-11T20:08:45.217589Z",
     "shell.execute_reply": "2024-04-11T20:08:45.216318Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.210625Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def return_prediction(model, text):\n",
    "    prediction = model.predict([text])[0]\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.219816Z",
     "iopub.status.busy": "2024-04-11T20:08:45.219557Z",
     "iopub.status.idle": "2024-04-11T20:08:45.251450Z",
     "shell.execute_reply": "2024-04-11T20:08:45.250400Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.219798Z"
    }
   },
   "outputs": [],
   "source": [
    "model = joblib.load(\"web_api/moment_predictor.joblib\") # load the saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.252703Z",
     "iopub.status.busy": "2024-04-11T20:08:45.252416Z",
     "iopub.status.idle": "2024-04-11T20:08:45.265116Z",
     "shell.execute_reply": "2024-04-11T20:08:45.262912Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.252684Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'affection'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"I love my students!\"\n",
    "return_prediction(model, text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.267626Z",
     "iopub.status.busy": "2024-04-11T20:08:45.267137Z",
     "iopub.status.idle": "2024-04-11T20:08:45.279647Z",
     "shell.execute_reply": "2024-04-11T20:08:45.277155Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.267600Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'enjoy_the_moment'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"ML is fun\"\n",
    "return_prediction(model, text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.283025Z",
     "iopub.status.busy": "2024-04-11T20:08:45.281967Z",
     "iopub.status.idle": "2024-04-11T20:08:45.294816Z",
     "shell.execute_reply": "2024-04-11T20:08:45.292870Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.282971Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'achievement'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"ML is hard\"\n",
    "return_prediction(model, text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function appears in the `app.py` that we'll be using shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Setting up a directory structure and environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We need a specific directory structure to help us easily deploy our machine learning model. \n",
    "- This is already set up in this repo.\n",
    "\n",
    "```\n",
    "├── web_api\n",
    "│   └── moment_predictor.joblib  # this is the machine learning model we have built locally\n",
    "│   └── app.py  # the file that defines our flask API\n",
    "│   └── Procfile  # required to help start flask app\n",
    "│   └── requirements.txt  # file containing required packages\n",
    "│   \n",
    "└── web_application\n",
    "    └── moment_predictor.joblib  # this is the machine learning model we have built locally\n",
    "    └── app.py  # the file that defines our flask API\n",
    "    └── Procfile  # required to help start flask app\n",
    "    └── requirements.txt  # file containing required packages\n",
    "    └── templates  # contains HTML templates to help us build the web application\n",
    "    │   └── style.css  # css template to be used in web application\n",
    "    └── static  # this subdirectory contains CSS style sheets\n",
    "        └── home.html  # html template to be used in web application\n",
    "        └── prediction.html  # html template to be used in web application\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have **two options** for deploying our moment prediction model. We can:\n",
    "\n",
    "1. Build a **web application (app)** with a HTML user-interface that interacts directly with our model.\n",
    "2. Develop a **RESTful** (REST stands for REpresentational State Transfer) **web API** that accepts HTTP requests in the form of input data and returns a prediction.\n",
    "\n",
    "We'll explore both options below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Building and deploying a web app\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |        ❓      |                          |\n",
    "| API  |                          |                          |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Flask can create entire web applications.\n",
    "- We need to link our code to some html and css to create our web application.\n",
    "- We will use Flask to create a html form, accept data submitted to the form, and return a prediction using the submitted data. \n",
    "- Again, I won't go into too much detail here, but we can open up `web_application/` and take a quick look."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- If you want to learn more about Flask, see:\n",
    "  - [Flask tutorial video series by Corey Schafer](https://www.youtube.com/playlist?list=PL-osiE80TeTs4UjLw5MM6OjgkjFeUxCYH)\n",
    "  - [Flask docs](https://flask.palletsprojects.com/en/1.1.x/)\n",
    "  - [Flask tutorial by Miguel Grinberg](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's try `web_application/app.py` that handles this part.\n",
    "- We can open it up here in Jupyter and take a look.\n",
    "- If we run `python app.py` we'll bring it to life."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Web app on local server\n",
    "\n",
    "1. Go to the terminal.\n",
    "1. Navigate to the `web_application` directory.\n",
    "1. Activate the `cpsc330` environment: `conda activate cpsc330`\n",
    "1. Run the following to make the app alive: `python app.py`.\n",
    "    - If you get an error, you may need to install those extra packages and make sure you have the environment loaded.\n",
    "1. Now you should be able to access the app at: `http://127.0.0.1:5000/` or `http://localhost:5000/`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Web app on a real server\n",
    "\n",
    "- If you want people to use your app/model, you would probably want to put it on a real server and not your laptop so that it's live all the time. \n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs)|\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |       ❓       |\n",
    "| API  |                          |                          |\n",
    "\n",
    "- We'll use [Render](https://render.com/) for this. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Render set-up (I already did these):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "1. Go to [render](https://dashboard.render.com/), log-in\n",
    "2. Click on \"New\" and create a web service.  \n",
    "\n",
    "![](../img/render-new-app.png)\n",
    "<!-- <img src=\"img/render-new-app.png\" width=\"600\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Demo of creating a new web service\n",
    "- Create an associated GitHub repository. \n",
    "- I've put the contents from `web_application` folder [in this repository](https://github.com/kvarada/ml-deployment-example).\n",
    "- Then you connect that repository to your new web service in render. \n",
    "- Once you fill in all the information, it'll create an environment for you using `requirement.txt` and if everything goes well, your web service will be live.\n",
    "- You'll see a link at the top which you can share with your friends. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Try it out for your own machine learning applications! \n",
    "- This is nice! If you develop a model and you want your friends to try it out without installing anything on their local computers, you can do this.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### API on the localhost \n",
    "\n",
    "- Often you want other people to be able to use your models in their applications.\n",
    "- We can do this by creating an **API**. \n",
    "- If you don't know what an API is, that's OK.\n",
    "  - For our purposes, it's something that exists at a particular address, that can accept information and return information.\n",
    "  - Sort of like a function but not Python-specific and potentially accessible by anyone on the internet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |        ✅       |\n",
    "| API  |       ❓      |                          |\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Go to the terminal. \n",
    "- Navigate to `web_api` folder in this repo.\n",
    "- Run the following to make the api alive: `python app.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that for more complex applications, you may choose to containerize everything in a Docker container to deploy to render)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sending a request to the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can use the exclamation mark ! to execute some operating system commands directly from within  Jupyter notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.296343Z",
     "iopub.status.busy": "2024-04-11T20:08:45.296080Z",
     "iopub.status.idle": "2024-04-11T20:08:45.427293Z",
     "shell.execute_reply": "2024-04-11T20:08:45.426044Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.296322Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is an example line to get printed out\n"
     ]
    }
   ],
   "source": [
    "!echo This is an example line to get printed out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.428672Z",
     "iopub.status.busy": "2024-04-11T20:08:45.428418Z",
     "iopub.status.idle": "2024-04-11T20:08:45.562472Z",
     "shell.execute_reply": "2024-04-11T20:08:45.559871Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.428651Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Procfile  default.profraw\t   requirements.txt\n",
      "app.py\t  moment_predictor.joblib  runtime.txt\n"
     ]
    }
   ],
   "source": [
    "!ls web_api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We have a RESTful (REST stands for REpresentational State Transfer) web API that accepts HTTP requests in the form of input data and returns a prediction.\n",
    "- Now you can send requests to the API and get predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T20:08:45.566558Z",
     "iopub.status.busy": "2024-04-11T20:08:45.565065Z",
     "iopub.status.idle": "2024-04-11T20:08:45.813906Z",
     "shell.execute_reply": "2024-04-11T20:08:45.812735Z",
     "shell.execute_reply.started": "2024-04-11T20:08:45.566519Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"achievement\"\n"
     ]
    }
   ],
   "source": [
    "!curl -d '{\"text\":\"The students will be submitting their last homework assignment today!\"}' \\\n",
    "      -H \"Content-Type: application/json\" \\\n",
    "      -X POST http://localhost:5000/predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `curl` (stands for client URL) is a tool for transferring data using various network protocols.\n",
    "\n",
    "Here's what each part does in the command: \n",
    "\n",
    "- `-d` $\\rightarrow$ allows you to send a POST request with the specified data.\n",
    "- `'{\"text\":\"The students will be submitting their last homework assignment soon!\"}'` $\\rightarrow$ the JSON data you're sending.\n",
    "- `-H` $\\rightarrow$ specifies a custom header, in this case, Content-Type: application/json to tell the server you're sending JSON.\n",
    "- `-X POST` $\\rightarrow$ explicitly sets the HTTP method to POST.\n",
    "- `http://localhost:5000/predict` $\\rightarrow$ the URL where you're sending the request.\n",
    "\n",
    "Make sure you have `app.py` running at port 5000 when you execute the curl command. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so we have a working API running on localhost, but we don't want to host this service on my laptop!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Deploying the API on a server\n",
    "\n",
    "- We now want to deploy it on a \"real\" server so others can send it requests. \n",
    "- We will use render to deploy our app but you could also use other services such as AWS.\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |        ✅       |\n",
    "| API  |       ✅      |                  ❓        |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Follow the instructions [here](https://render.com/docs/api) to try the API on server. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short summary:\n",
    "\n",
    "#### You will need a github repository with the app\n",
    "\n",
    "- You can create it as a public or private repository on your own github.\n",
    "- Copy the web_application folder from https://github.com/UBC-CS/cpsc330-2024s/tree/main/lectures/mehrdad in the repo and push.\n",
    "- You can also copy web_api if that's the one you want to run.\n",
    "\n",
    "You can then follow this (very short) tutorial!\n",
    "\n",
    "1. Go to render, log-in, and click \"New > Web Service\".\n",
    "2. Link your github repository.\n",
    "3. Choose a unique name for your app. (I chose `class-demo-330`.)\n",
    "   - Environment Python\n",
    "   - Build Command pip install -r requirements.txt\n",
    "   - Start Command gunicorn app:app\n",
    "   - Root Directory web_application (if that's where app.py is)\n",
    "   - Add environment variable PYTHON_VERSION 3.10.9\n",
    "   - Create app.\n",
    "\n",
    "#### Testing the web application\n",
    "\n",
    "I have done this already and our app should be live at this link!\n",
    "\n",
    "https://class-demo-330.onrender.com\n",
    "\n",
    "- Try it out!\n",
    "- This is nice! If you develop a model and you want your friends to try it out without installing anything on their local computers, you can do this.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "All done!\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|:---:|:---:|:---:|\n",
    "| app  |              ✅            |        ✅       |\n",
    "| API  |       ✅      |                  ✅        |\n",
    "\n",
    "\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Discussion\n",
    "\n",
    "- There are many ways to deploy a model; a RESTful API is very common and convenient. \n",
    "- As you can see, a simple deployment is fairly straightward. \n",
    "- However, there may be other considerations such as:\n",
    "  - Privacy/security\n",
    "  - Scaling to large number of requests\n",
    "  - Error handling\n",
    "  - Real-time / speed\n",
    "  - Low-resource environments (e.g. edge computing)\n",
    "  - etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Course review / conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We started with these learning objectives. \n",
    "\n",
    "1. Identify problems that may be addressed with machine learning.\n",
    "2. Select the appropriate machine learning tool for a problem.\n",
    "3. Transform data of various types into usable features.\n",
    "4. Apply standard tools implementing supervised and unsupervised learning techniques.\n",
    "5. Describe core differences between training, validation, and testing regimes.\n",
    "6. Effectively communicate the results of a machine learning pipeline.\n",
    "7. Be realistic about the limitations of individual approaches and machine learning as a whole. \n",
    "8. Identify and avoid scenarios in which training and testing data are accidentally mixed (the \"Golden Rule\").\n",
    "9. Employ good habits for applying ML, such as starting an analysis with a baseline estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How did we do? \n",
    "- Hopefully OK.\n",
    "- Another important part of this is create reproducible workflows and pipelines. \n",
    "- There is a [new course](https://ubc-dsci.github.io/dsci-310-student/) on this topic. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What did we cover?\n",
    "\n",
    "I see the course roughly like this (not necessarily in order):\n",
    "\n",
    "Part 1: Supervised learning on tabular data\n",
    "\n",
    "- Overfitting, train/validation/test/deployment, cross-validation\n",
    "- Feature preprocessing, pipelines, imputation, OHE, etc\n",
    "- The Golden Rule, various ways to accidentally violate it\n",
    "- Classification metrics: confusion matrix, precision/recall, ROC, AUC\n",
    "- Regression metrics: MSE, MAPE\n",
    "- Feature importances, feature selection\n",
    "- Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- A bunch of models: \n",
    "  - baselines\n",
    "  - linear models (ridge, logistic regression)\n",
    "  - KNNs and RBF SVMs\n",
    "  - tree-based models (random forest, gradient boosted trees)\n",
    "  - ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Part 2: Other data types (non-tabular)\n",
    "\n",
    "- Clustering: K-Means, DBSCAN\n",
    "- Recommender systems \n",
    "- Computer vision with pre-trained deep learning models (high level)\n",
    "- Language data, text preprocessing, embeddings, topic modeling\n",
    "- Time series\n",
    "- Right-censored data / survival analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Part 3: Communication and Ethics\n",
    "\n",
    "- Communicating your results \n",
    "- Ethics for ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ❓❓ Questions for you\n",
    "\n",
    "For each of the scenarios below \n",
    "- Identify if ML is a good solution for a problem.\n",
    "- If yes\n",
    "    - Frame the problem to a ML problem.\n",
    "    - Discuss what kind of features you would need to effectively solve the problem\n",
    "    - What would be a reasonable baseline? \n",
    "    - Which model would be a suitable model for the given scenario? \n",
    "    - What would be the appropriate success metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|     App           | Goal | \n",
    "|-------------------|--------------------------|\n",
    "| QueuePredictor app    |  Inform callers how long they'll wait on hold given the current call volume|\n",
    "| To-doList App    | Keep track of the tasks that a user inputs and organize them by date | \n",
    "| SegmentSphere App | To segment customers to tailor marketing strategies based on purchasing behavior |\n",
    "| Video app\t        | Recommend useful videos|\n",
    "| Dining app        | Identify cuisine by a restaurant's menu|\n",
    "| Weather app       | Calculate precipitation in six hour increments for a geographic region  |\n",
    "| EvoCarShare app   | Calculate number of car rentals in four increaments at a particular Evo parking spot |\n",
    "| Pharma app| Understand the effect of a new drug on patient survival time |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some key takeaways\n",
    "\n",
    "Some useful guidelines:\n",
    "\n",
    "- Do train-test split right away and only once\n",
    "- Don't look at the test set until the end\n",
    "- Don't call `fit` on test/validation data\n",
    "- Use pipelines\n",
    "- Use baselines\n",
    "- Do not be overconfident about your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Difference between Statistics and Machine Learning\n",
    "- There is no clear boundary. \n",
    "- But loosely\n",
    "    - in statistics the emphasis is on inference (usually framed as hypothesis testing)\n",
    "    - in machine learning the emphasis is on prediction and generalization. We are primarily interested in using our model on new unseen examples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recipe to approach a supervised learning problem with tabular data \n",
    "\n",
    "#### Understanding the problem\n",
    "\n",
    "1. Have a long conversation with the stakeholder(s) who will be using your pipeline. \n",
    "2. Have a long conversation with the person(s) who collected the data.\n",
    "3. Think about \n",
    "    - what do you want to accomplish?\n",
    "    - whether you really need to use machine learning for the problem or not\n",
    "    - how do you want to frame the problem as a supervised machine learning problem? What do you want to predict? \n",
    "    - how do you plan to measure your success? \n",
    "    - is there a baseline? \n",
    "    - is there any operating point? \n",
    "    - what will it buy the stakeholders if you improve over their current baseline?  \n",
    "4. Think about the ethical implications - are you sure you want to do this project? If so, should ethics guide your approach?\n",
    "\n",
    "\n",
    "#### Initial analysis, EDA, preprocessing\n",
    "1. Random train-test split with fixed random seed; do not touch the test data until Step 16.\n",
    "2. Exploratory data analysis, outlier detection.\n",
    "3. Choose a scoring metric -> higher values should make you & your stakeholders happier.\n",
    "4. Feature engineering --> Create new features which might be useful to solve the prediction problem. This is typically a time-consuming process. Also, sometimes you get an idea of good features only after looking at your model results. So it's an iterative process. \n",
    "5. Fit a baseline model, e.g. `DummyClassifier` or `DummyRegressor`.\n",
    "6. Create a preprocessing pipeline. \n",
    "7. (Optional) Incorporate feature selection. \n",
    "\n",
    "#### Model building \n",
    "1. Try a linear model, e.g. `LogisticRegression` for classification or `Ridge`; tune hyperparameters with CV.\n",
    "2. Try other sensible model(s), e.g. LightGBM; tune hyperparameters with CV.\n",
    "3. For each model, look at sub-scores from the folds of cross-validation to get a sense of \"error bars\" on the scores.\n",
    "4. Pick a few reasonable models. Best CV score is a reasonable metric, though you may choose to favour simpler models.\n",
    "5. Carry out hyperparameter optimization for these models, paying attention to whether you are not susceptible to optimization bias. \n",
    "6. Try averaging and stacking to examine whether you improve the results over  your best-performing models.  \n",
    "\n",
    "#### Model transparency and interpretation \n",
    "1. Look at feature importances.\n",
    "2. (optional) Perform some more diagnostics like confusion matrix for classification, or \"predicted vs. true\" scatterplots for regression.\n",
    "3. (optional) Try to calibrate the uncertainty/confidence outputted by your model.\n",
    "4. Test set evaluation.\n",
    "5. Model transparency. Take a random sample from the test data and try to explain your predictions using SHAP plots. \n",
    "6. Question everything again: validity of results, bias/fairness of trained model, etc.\n",
    "7. Concisely summarize your results and discuss them with stakeholders.\n",
    "8. (optional) Retrain on all your data.\n",
    "9. Deployment & integration.\n",
    "\n",
    "PS: the order of steps is approximate, and some steps may need to be repeated during prototyping, experimentation, and as needed over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "The order of steps is approximate, and some steps may need to be repeated during prototyping, experimentation, and as needed over time. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What next? \n",
    "\n",
    "If you want to further develop your machine learning skills: \n",
    "- Practice! \n",
    "    - Work on your own projects\n",
    "- Work hard and be consistent.   \n",
    "- If you are interested in research in machine learning\n",
    "    - Take CPSC 340. If you do not have the required prereqs you can try to audit it. \n",
    "    - Get into the habit of reading papers and replicating results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 330 vs. 340\n",
    "\n",
    "- We are hoping lots of people will take both courses.\n",
    "- There is some overlap but not a crazy amount.\n",
    "- If you want to learn how these methods work under the hood, CPSC 340 will give you a lot of that, such as:\n",
    "  - Implementing `Ridge.fit()` from scratch\n",
    "  - Mathematically speaking, what is `C` in `LogisticRegression`?\n",
    "  - How fast do these algorithms run in terms of the number of rows and columns of your dataset? \n",
    "  - Etc.\n",
    "- There are also a bunch of other methods covered. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusion & farewell\n",
    "\n",
    "That's all, folks. We made it! Good luck on your final exam! When you get a chance, please let me know what worked for you and what didn't work for you in this course.   \n",
    "\n",
    "![](../img/eva-congrats.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}