Deep Learning Models - with windows.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SniffNets\n",
    "\n",
    "This notebook contains the implementation of the SniffNets models created for clasifying the codification of signals deteced from artificial noses. The experiments in this notebook were used in the article [Deep learning models for classification of gases detected by sensor arrays of artificial nose](https://sol.sbc.org.br/index.php/eniac/article/view/9339).\n",
    "\n",
    "Where the data features is divided in sliding windows, which is a rapid detection technique stated in the article [Wine quality rapid detection using a compact electronic nose system: application focused on\n",
    "spoilage thresholds by acetic acid](https://arxiv.org/pdf/2001.06323.pdf)\n",
    "\n",
    "* Disclaimer: Unfortunately the Coffee dataset is not publicly available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the following code if you are using google colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this block it you are using google colaboratory and desire to save the results in\n",
    "# google drive\n",
    "from google.colab import drive\n",
    "drive.mount(\"/content/drive\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/IsmaelCesar/SniffNets.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"SniffNets/\")\n",
    "import data_loading\n",
    "data_loading.DATA_FOLDER = os.path.join(\"SniffNets/\",data_loading.DATA_FOLDER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining and running experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from sklearn.utils import shuffle\n",
    "from data_loading import load_dataset, split_datasamples_by_sensors, standardize_data\n",
    "from data_loading import dataset_classes_number, dataset_wine_classes_number\n",
    "from experiment_procedures import resettv, get_dataset_names_and_indexes, get_wine_dataset_names_and_indexes\n",
    "from models import sniffnet, sniffresnet, sniffmultinose\n",
    "from evaluation import evaluate_model_windows, save_results_into_filesystem_windows\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.optimizers import SGD\n",
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining experiments parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 10\n",
    "epochs = 20\n",
    "lr = .001\n",
    "parameters = resettv()\n",
    "\n",
    "# Experiment results to be saved\n",
    "experiment_folder = \"put the directory here\"\n",
    "save_results = False\n",
    "time_estimate_list_titles = [[\"model type\",\n",
    "                              \"datset and sub dataset name\",\n",
    "                              \"total_execution_time\",\n",
    "                              \"windows_processed\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deffining auxiliary procedures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_windowed_experiment(batch_size, epochs, lr, parameters, \n",
    "                            save_results=True, read_wine=False):\n",
    "    global experiment_folder\n",
    "    list_of_keys = list(parameters.keys())\n",
    "\n",
    "    if read_wine:\n",
    "        # Creating indexes for each dataset being loaded\n",
    "        names_list = list(get_wine_dataset_names_and_indexes().keys())\n",
    "    else: \n",
    "        datasets_names = get_dataset_names_and_indexes()\n",
    "\n",
    "    names_list = list(datasets_names.keys())\n",
    "    \n",
    "    # Taking the parameters for fonollosa dataset as a test case\n",
    "    # Iterating over model type, where the all the model types created are the normal ConvNet, Resnet an FusionNEt\n",
    "    # Where for the fusion net the data needs a special treatment\n",
    "    models_names = [(0, \"SniffConvNet\"), (1, \"SniffResnet\"), (2, \"SniffMultinose\")]    \n",
    "    \n",
    "    for (model_type, m_name) in models_names:\n",
    "        print(\"\\n\\tUsin \" + m_name + \" architechture\\n\")\n",
    "        model_folder = m_name + \"/\"\n",
    "        # Interating over dataset names\n",
    "        for i, name in enumerate(names_list):\n",
    "            print(\"\\n\\tDataset \" + name + \"\\n\")\n",
    "            sub_set_index = datasets_names[name]\n",
    "            # Iterating over subset indexes\n",
    "            for ss_idx in sub_set_index:\n",
    "                print(\"\\n\\n ds_name:\"+name+\"\\n\\n\")\n",
    "                print(\"\\n\\n ds_idx:\"+str(ss_idx)+\"\\n\\n\")\n",
    "                f_params = parameters[name]\n",
    "                # Load dataset\n",
    "                (data, labels, n_classes,\n",
    "                 dataset_name, sub_dataset_name, input_shape) = load_dataset(name, ss_idx,\n",
    "                                                                             read_wine_datasets=read_wine)\n",
    "                time_estimate_list = []\n",
    "                toc = time.time()\n",
    "                n_windows_processed = 0\n",
    "                data, labels = shuffle(data, labels)\n",
    "                for final_measurement in range(f_params['start_value'], f_params['end_value'], f_params['step']):\n",
    "                    print(\"\\n\\n\\t Window\", final_measurement)\n",
    "                    print(\"\\n\\n\")\n",
    "                    print(\"\\n\\tSubset \" + sub_dataset_name + \"\\n\")\n",
    "                    train_data, test_data, train_labels, test_labels = train_test_split(\n",
    "                        data[:, f_params['ini_value']:final_measurement, :],\n",
    "                        labels, test_size=0.2)\n",
    "                    # Normalizing data\n",
    "                    train_data, test_data = standardize_data(train_data, test_data, test_data.shape[1:])\n",
    "\n",
    "                    train_labels = to_categorical(train_labels, n_classes)\n",
    "                    test_labels = to_categorical(test_labels, n_classes)\n",
    "\n",
    "                    if model_type == 2:\n",
    "                        # Reshapes data if usign the SniffMultinose\n",
    "                        train_data = data_set_reshaped(train_data)\n",
    "                        test_data = data_set_reshaped(test_data)\n",
    "                        input_shape = train_data[0].shape\n",
    "                        train_data = split_datasamples_by_sensors(train_data)\n",
    "                        test_data = split_datasamples_by_sensors(test_data)\n",
    "\n",
    "                    # defining model\n",
    "                    model = None\n",
    "                    if model_type == 0:\n",
    "                        model = sniffnet(train_data.shape[1:], n_classes)\n",
    "                    elif model_type == 1:\n",
    "                        model = sniffresnet(train_data.shape[1:], n_classes)\n",
    "                    elif model_type == 2:\n",
    "                        model = sniffmultinose(input_shape, n_classes)\n",
    "                    elif model_type == 3:\n",
    "                        model = get_svm()\n",
    "                    print(\"Model \" + m_name + \" has been created\")\n",
    "\n",
    "                    model.compile(loss=\"categorical_crossentropy\",\n",
    "                                  optimizer= SGD(lr=lr, momentum=.9),\n",
    "                                  metrics=['accuracy'])\n",
    "                    H = model.fit(train_data, train_labels, batch_size=batch_size, epochs=epochs, verbose=1,\n",
    "                                  validation_data=(test_data, test_labels))\n",
    "\n",
    "                    n_windows_processed += 1\n",
    "                    \n",
    "                    evaluate_model_windows(test_data, test_labels, batch_size, model, epochs, H, \n",
    "                                           experiment_folder, dataset_name, sub_dataset_name, model_folder,\n",
    "                                           window_size=str(final_measurement), save_results=save_results)\n",
    "                    \n",
    "                tic = time.time()\n",
    "                time_estimate_set_names = name+\" \"+sub_dataset_name\n",
    "                total_estimate = tic - toc\n",
    "                time_estimate_list.append([m_name,time_estimate_set_names,\n",
    "                                           total_estimate, \n",
    "                                           n_windows_processed])\n",
    "                if save_results: \n",
    "                    write_csv(experiment_folder,\"time_estimates_wt_fon_tgm_coff.csv\",\n",
    "                              time_estimate_list,\n",
    "                              mode='a+')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Runnig experiments for Fonollosa, Windtunnel and Turbulent Gas Mixtures datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if save_results:\n",
    "    write_csv(experiment_folder,\"time_estimates_wt_fon_tgm.csv\", time_estimate_list_titles)\n",
    "\n",
    "run_windowed_experiment(batch_size, epochs, lr, parameters, save_results=save_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running experiments for the Wine dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if save_results:\n",
    "    write_csv(experiment_folder,\"time_estimates_wt_fon_tgm.csv\", time_estimate_list_titles)\n",
    "\n",
    "run_windowed_experiment(batch_size, epochs, lr, parameters, save_results=save_results, read_wine=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}