keras-cifar10-training-w-mlrun.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow.keras cifar10 Training with MLRun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_1\"></a>\n",
    "## 1. Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_2\"></a>\n",
    "## 2. Setup MLRun project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-05-24 15:08:47,127 [info] loaded project keras-cifar10-jovyan from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "# Initialize the MLRun environment and save the project name and artifacts path:\n",
    "project_name, artifact_path = mlrun.set_environment(\n",
    "    project=\"keras-cifar10\", user_project=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_3\"></a>\n",
    "## 3. Traing the Model Using MLRun\n",
    "\n",
    "Now, we will create our function and run it multiple times for our hyperparameters tunning. Once the function is running, you can click on the link and see the logs collected into **MLRun**. For **Tensorboard** you will need to have a tensorboard running at the `/User` directory. We can train it:\n",
    "1. **Locally** - To run locally, set the `local` parameter to `True`.\n",
    "2. As a **Job** - To run as a job, set the `kind` parameter to `\"job\"`.\n",
    "3. As a **MPIJob** - To run as a mpijob, set the `kind` parameter to `\"mpijob\"`. MPIJob will setup horovod automatically as noted above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "local = False # True or False\n",
    "kind = \"job\"  # \"job\" or \"mpijob\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f4dc16fbee0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the mlrun function:\n",
    "\n",
    "training_function = mlrun.code_to_function(\n",
    "    name='cifar10-trainer',\n",
    "    project=project_name,\n",
    "    filename=\"keras-cifar10-original-train-code-with-2-added-lines.ipynb\",\n",
    "    kind=kind,\n",
    "    image = 'mlrun/ml-models:0.10.0'\n",
    ")\n",
    "\n",
    "# Setup further configurations for deployment:\n",
    "training_function.apply(mlrun.platforms.auto_mount())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the training job with fixed set of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-05-24 15:08:58,380 [info] starting run cifar10-simple-trainer uid=bbb7e5d1f7f047ca9868621c1c255652 DB=http://mlrun-api:8080\n",
      "> 2022-05-24 15:08:59,040 [info] Job is running in the background, pod: cifar10-simple-trainer-g5lhs\n",
      "batch size    === >>> 64\n",
      "learning rate === >>> 0.001\n",
      "epochs        === >>> 5\n",
      "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
      "170500096/170498071 [==============================] - 66s 0us/step/170498071 [=>............................] - ETA: 1:7712/170498071 [====>.........................] - ETA: 1:120/170498071 [========>.....................] - ETA: 1:A: 51: 37 2\n",
      "170508288/170498071 [==============================] - 66s 0us/step\n",
      "2022-05-24 15:12:26.016144: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-05-24 15:12:26.044731: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "2022-05-24 15:12:26.586234: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "Epoch 1/5\n",
      "782/782 [==============================] - 237s 297ms/step - loss: 1.7993 - accuracy: 0.3758 - val_loss: 1.5639 - val_accuracy: 0.4542....] - ETA: 2:36 - loss: 2.1683 - accuracy: 0..33.........] - ETA: 1:00 - loss: 1.9124 - accuracy: 0\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 200s 255ms/step - loss: 1.2893 - accuracy: 0.5347 - val_loss: 1.2868 - val_accuracy: 0.5650loss: 1.3875 - accuracy:y: 0.52\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 190s 242ms/step - loss: 1.1099 - accuracy: 0.6055 - val_loss: 1.2775 - val_accuracy: 0.5610loss: 1.1596 - accura\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 193s 246ms/step - loss: 0.9908 - accuracy: 0.6534 - val_loss: 1.2946 - val_accuracy: 0.5792loss: 1.0208 - accura\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 196s 251ms/step - loss: 0.9185 - accuracy: 0.6796 - val_loss: 1.2987 - val_accuracy: 0.5799loss: 0.9296 - accura\n",
      "> 2022-05-24 15:30:25,677 [info] best iteration=1, used criteria max.validation_accuracy\n",
      "> 2022-05-24 15:30:26,211 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>keras-cifar10-jovyan</td>\n",
       "      <td><div title=\"bbb7e5d1f7f047ca9868621c1c255652\"><a href=\"http://localhost:30060/projects/keras-cifar10-jovyan/jobs/monitor/bbb7e5d1f7f047ca9868621c1c255652/overview\" target=\"_blank\" >...1c255652</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 24 15:09:21</td>\n",
       "      <td>completed</td>\n",
       "      <td>cifar10-simple-trainer</td>\n",
       "      <td><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=jovyan</div><div class=\"dictlist\">mlrun/client_version=0.10.0</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">batch_size=64</div><div class=\"dictlist\">epochs=5</div><div class=\"dictlist\">lr=0.0010000000474974513</div><div class=\"dictlist\">training_loss=0.81195068359375</div><div class=\"dictlist\">training_accuracy=0.67181396484375</div><div class=\"dictlist\">validation_loss=1.2986965971632887</div><div class=\"dictlist\">validation_accuracy=0.5799000407941044</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/training_loss.html\">training_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/training_accuracy.html\">training_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/validation_loss.html\">validation_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/validation_accuracy.html\">validation_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/loss_summary.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/accuracy_summary.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/1/lr_values.html\">lr_values.html</div><div title=\"/home/jovyan/data/1\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/iteration_results.csv\">iteration_results</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result448e5a89\" title=\"/files/data/parallel_coordinates.html\">parallel_coordinates</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result448e5a89-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result448e5a89-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result448e5a89\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result448e5a89-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"http://localhost:30060/projects/keras-cifar10-jovyan/jobs/monitor/bbb7e5d1f7f047ca9868621c1c255652/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-05-24 15:30:36,460 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# Run it:\n",
    "grid_params = {\n",
    "    \"batch_size\": [64], \n",
    "    \"lr\": [1e-3],\n",
    "    \"epochs\": [5]\n",
    "}\n",
    "\n",
    "task = mlrun.new_task(\"cifar10-simple-trainer\").with_hyper_params(grid_params,selector=\"max.validation_accuracy\")\n",
    "training_run = training_function.run(task, \n",
    "                                     handler=\"train\",\n",
    "                                     local=False,\n",
    "                                     watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the training job with hyperparameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-05-24 15:30:44,742 [info] starting run cifar10-hp-trainer uid=86337f5c1f06423bb27aa41a2e52ff61 DB=http://mlrun-api:8080\n",
      "> 2022-05-24 15:30:44,968 [info] Job is running in the background, pod: cifar10-hp-trainer-s9kxx\n",
      "batch size    === >>> 64\n",
      "learning rate === >>> 0.01\n",
      "epochs        === >>> 10\n",
      "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
      "170500096/170498071 [==============================] - 56s 0us/step.................] - ETA: 59....................] - ETA: 54...................] - ETA: 41==>...............] - ETA: 29====>............] - ETA: 22========>.......] - ETA: 12\n",
      "170508288/170498071 [==============================] - 56s 0us/step\n",
      "2022-05-24 15:32:20.012030: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-05-24 15:32:20.014045: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "2022-05-24 15:32:20.208497: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "Epoch 1/10\n",
      "782/782 [==============================] - 241s 306ms/step - loss: 1.6707 - accuracy: 0.3954 - val_loss: 1.4942 - val_accuracy: 0.4796:22 - loss: 1.9630 - accuracy: 07s - loss: 1.7614 - accuracy: 0.365================>.....] - ETA: 31s - loss: 1.7174 - accuracy: 0.380=================>.] - ETA: 5s - loss: 1.6775 - accuracy: 0.39\n",
      "Epoch 2/10\n",
      "782/782 [==============================] - 211s 270ms/step - loss: 1.2582 - accuracy: 0.5492 - val_loss: 1.2933 - val_accuracy: 0.5763loss: 1.3262 - accuracy:ss: 1.2680 - accuracy: 0.5461.2602 - accuracy: 0.54\n",
      "Epoch 3/10\n",
      "782/782 [==============================] - 210s 268ms/step - loss: 1.0978 - accuracy: 0.6135 - val_loss: 1.3259 - val_accuracy: 0.5812loss: 1.1363 - accuracy:uracy: 0.611 0.61\n",
      "Epoch 4/10\n",
      "782/782 [==============================] - 205s 262ms/step - loss: 1.0046 - accuracy: 0.6527 - val_loss: 0.9117 - val_accuracy: 0.6889loss: 1.0308 - accuracy: - accuracy: 0.650uracy: 0.65\n",
      "Epoch 5/10\n",
      "782/782 [==============================] - 203s 260ms/step - loss: 0.9432 - accuracy: 0.6740 - val_loss: 0.8838 - val_accuracy: 0.6973loss: 0.9594 - accuracy:9460 - accuracy: 0.671ccuracy: 0.67\n",
      "Epoch 6/10\n",
      "782/782 [==============================] - 207s 265ms/step - loss: 0.8939 - accuracy: 0.6907 - val_loss: 1.0603 - val_accuracy: 0.6570loss: 0.8967 - accura\n",
      "Epoch 7/10\n",
      "782/782 [==============================] - 197s 252ms/step - loss: 0.8588 - accuracy: 0.7063 - val_loss: 0.9376 - val_accuracy: 0.6822loss: 0.8621 - accura\n",
      "Epoch 8/10\n",
      "782/782 [==============================] - 196s 250ms/step - loss: 0.8243 - accuracy: 0.7180 - val_loss: 0.8326 - val_accuracy: 0.7211loss: 0.8356 - accura\n",
      "Epoch 9/10\n",
      "782/782 [==============================] - 204s 261ms/step - loss: 0.7962 - accuracy: 0.7267 - val_loss: 0.9252 - val_accuracy: 0.6992loss: 0.8010 - accuracy: accuracy: 0.725acy: 0.72\n",
      "Epoch 10/10\n",
      "782/782 [==============================] - 200s 256ms/step - loss: 0.7809 - accuracy: 0.7325 - val_loss: 0.6823 - val_accuracy: 0.7732loss: 0.7739 - accuracy: 0.73\n",
      "batch size    === >>> 128\n",
      "learning rate === >>> 0.01\n",
      "epochs        === >>> 10\n",
      "Epoch 1/10\n",
      "391/391 [==============================] - 213s 541ms/step - loss: 1.6963 - accuracy: 0.3860 - val_loss: 1.7311 - val_accuracy: 0.4243 - ETA: 4s - loss: 1.7032 - accuracy: 0.38\n",
      "Epoch 2/10\n",
      "391/391 [==============================] - 187s 478ms/step - loss: 1.2885 - accuracy: 0.5360 - val_loss: 1.5060 - val_accuracy: 0.5291\n",
      "Epoch 3/10\n",
      "391/391 [==============================] - 186s 476ms/step - loss: 1.1045 - accuracy: 0.6069 - val_loss: 1.2024 - val_accuracy: 0.6179\n",
      "Epoch 4/10\n",
      "391/391 [==============================] - 222s 567ms/step - loss: 0.9904 - accuracy: 0.6526 - val_loss: 1.0482 - val_accuracy: 0.6496- ETA: 5s - loss: 0.9910 - accuracy: 0.65\n",
      "Epoch 5/10\n",
      "391/391 [==============================] - 206s 527ms/step - loss: 0.9165 - accuracy: 0.6788 - val_loss: 0.9985 - val_accuracy: 0.67779176 - accuracy: 0.67\n",
      "Epoch 6/10\n",
      "391/391 [==============================] - 202s 517ms/step - loss: 0.8707 - accuracy: 0.6997 - val_loss: 0.8334 - val_accuracy: 0.7161ss: 0.8704 - accuracy: 0.69\n",
      "Epoch 7/10\n",
      "391/391 [==============================] - 203s 518ms/step - loss: 0.8288 - accuracy: 0.7144 - val_loss: 0.7735 - val_accuracy: 0.7427 4s - loss: 0.8288 - accuracy: 0.71\n",
      "Epoch 8/10\n",
      "391/391 [==============================] - 203s 520ms/step - loss: 0.8029 - accuracy: 0.7257 - val_loss: 0.7120 - val_accuracy: 0.7638ETA: 4s - loss: 0.8046 - accuracy: 0.72\n",
      "Epoch 9/10\n",
      "391/391 [==============================] - 201s 514ms/step - loss: 0.7737 - accuracy: 0.7351 - val_loss: 0.7322 - val_accuracy: 0.7593 4s - loss: 0.7752 - accuracy: 0.73\n",
      "Epoch 10/10\n",
      "391/391 [==============================] - 211s 541ms/step - loss: 0.7553 - accuracy: 0.7423 - val_loss: 0.7026 - val_accuracy: 0.7613.744========>.] - ETA: 5s - loss: 0.7552 - accuracy: 0.74\n",
      "batch size    === >>> 64\n",
      "learning rate === >>> 0.001\n",
      "epochs        === >>> 10\n",
      "Epoch 1/10\n",
      "782/782 [==============================] - 264s 335ms/step - loss: 1.8309 - accuracy: 0.3692 - val_loss: 1.6145 - val_accuracy: 0.4400:34 - loss: 2.2219 - accuracy: 0...] - ETA: 1:03 - loss: 1.9538 - accuracy: 0\n",
      "Epoch 2/10\n",
      "782/782 [==============================] - 274s 350ms/step - loss: 1.3301 - accuracy: 0.5223 - val_loss: 1.3561 - val_accuracy: 0.5305loss: 1.3980 - accuracy: 0] - ETA: 1:14 - loss: 1.3622 - accuracy: 0.5cy: 0.52\n",
      "Epoch 3/10\n",
      "782/782 [==============================] - 233s 298ms/step - loss: 1.1321 - accuracy: 0.5960 - val_loss: 1.2057 - val_accuracy: 0.5822loss: 1.1760 - accuracy: 0 accuracy: 0.587...] - ETA: 31s - loss: 1.1442 - accuracy: 0.591====>.] - ETA: 5s - loss: 1.1335 - accuracy: 0.59\n",
      "Epoch 4/10\n",
      "782/782 [==============================] - 249s 318ms/step - loss: 1.0250 - accuracy: 0.6396 - val_loss: 1.1674 - val_accuracy: 0.5953loss: 1.0422 - accuracy: 0] - ETA: 1:05 - loss: 1.0327 - accuracy: 0\n",
      "Epoch 5/10\n",
      "782/782 [==============================] - 221s 282ms/step - loss: 0.9395 - accuracy: 0.6732 - val_loss: 0.7844 - val_accuracy: 0.7249loss: 0.9626 - accuracy: 0 - accuracy: 0.669.....] - ETA: 30s - loss: 0.9459 - accuracy: 0.670====>.] - ETA: 5s - loss: 0.9401 - accuracy: 0.67\n",
      "Epoch 6/10\n",
      "782/782 [==============================] - 241s 308ms/step - loss: 0.8797 - accuracy: 0.6927 - val_loss: 0.7818 - val_accuracy: 0.7253loss: 0.8923 - accuracy: 0ccuracy: 0.689.] - ETA: 32s - loss: 0.8801 - accuracy: 0.692====>.] - ETA: 5s - loss: 0.8780 - accuracy: 0.69\n",
      "Epoch 7/10\n",
      "782/782 [==============================] - 238s 305ms/step - loss: 0.8434 - accuracy: 0.7080 - val_loss: 0.7533 - val_accuracy: 0.7402loss: 0.8434 - accuracy: 0] - ETA: 59s - loss: 0.8511 - accuracy: 0.70452 [========================>.....] - ETA: 33s - loss: 0.8469 - accuracy: 0.70561/782 [============================>.] - ETA: 5s - loss: 0.8442 - accuracy: 0.70\n",
      "Epoch 8/10\n",
      "782/782 [==============================] - 248s 318ms/step - loss: 0.7975 - accuracy: 0.7259 - val_loss: 0.6778 - val_accuracy: 0.7684loss: 0.8100 - accuracy: 0] - ETA: 1:00 - loss: 0.8041 - accuracy: 0.\n",
      "Epoch 9/10\n",
      "782/782 [==============================] - 250s 320ms/step - loss: 0.7646 - accuracy: 0.7386 - val_loss: 0.6651 - val_accuracy: 0.7718loss: 0.7679 - accuracy: 0] - ETA: 1:03 - loss: 0.7639 - accuracy: 0\n",
      "Epoch 10/10\n",
      "782/782 [==============================] - 243s 311ms/step - loss: 0.7481 - accuracy: 0.7434 - val_loss: 0.6333 - val_accuracy: 0.7854loss: 0.7591 - accuracy: 0] - ETA: 1:01 - loss: 0.7545 - accuracy: 0\n",
      "batch size    === >>> 128\n",
      "learning rate === >>> 0.001\n",
      "epochs        === >>> 10\n",
      "Epoch 1/10\n",
      "391/391 [==============================] - 254s 643ms/step - loss: 1.9054 - accuracy: 0.3486 - val_loss: 1.9829 - val_accuracy: 0.3153:03 - loss: 2.0266 - accuracy: 0.3\n",
      "Epoch 2/10\n",
      "391/391 [==============================] - 229s 586ms/step - loss: 1.4032 - accuracy: 0.4920 - val_loss: 1.3998 - val_accuracy: 0.50791.4330 - accuracy: 0.479 [============================>.] - ETA: 5s - loss: 1.4050 - accuracy: 0.49\n",
      "Epoch 3/10\n",
      "391/391 [==============================] - 253s 647ms/step - loss: 1.2136 - accuracy: 0.5672 - val_loss: 1.2091 - val_accuracy: 0.5922loss: 1.2332 - accuracy: 0.\n",
      "Epoch 4/10\n",
      "391/391 [==============================] - 333s 854ms/step - loss: 1.0889 - accuracy: 0.6169 - val_loss: 1.0030 - val_accuracy: 0.6503loss: 1.1048 - accuracy: 0.\n",
      "Epoch 5/10\n",
      "391/391 [==============================] - 223s 570ms/step - loss: 0.9960 - accuracy: 0.6475 - val_loss: 0.9678 - val_accuracy: 0.6720acy: 0.643==============>.] - ETA: 5s - loss: 0.9969 - accuracy: 0.64\n",
      "Epoch 6/10\n",
      "391/391 [==============================] - 214s 548ms/step - loss: 0.9196 - accuracy: 0.6781 - val_loss: 0.8282 - val_accuracy: 0.7162- ETA: 5s - loss: 0.9207 - accuracy: 0.67\n",
      "Epoch 7/10\n",
      "391/391 [==============================] - 236s 603ms/step - loss: 0.8739 - accuracy: 0.6959 - val_loss: 0.8477 - val_accuracy: 0.7135ss: 0.8761 - accuracy: 0.69481/391 [============================>.] - ETA: 5s - loss: 0.8750 - accuracy: 0.69\n",
      "Epoch 8/10\n",
      "391/391 [==============================] - 226s 577ms/step - loss: 0.8316 - accuracy: 0.7106 - val_loss: 0.8484 - val_accuracy: 0.71100.8323 - accuracy: 0.709 [============================>.] - ETA: 5s - loss: 0.8321 - accuracy: 0.71\n",
      "Epoch 9/10\n",
      "391/391 [==============================] - 220s 563ms/step - loss: 0.7998 - accuracy: 0.7216 - val_loss: 0.7640 - val_accuracy: 0.7436ccuracy: 0.720==================>.] - ETA: 5s - loss: 0.8000 - accuracy: 0.72\n",
      "Epoch 10/10\n",
      "391/391 [==============================] - 232s 593ms/step - loss: 0.7689 - accuracy: 0.7337 - val_loss: 0.7568 - val_accuracy: 0.74897692 - accuracy: 0.734============================>.] - ETA: 5s - loss: 0.7686 - accuracy: 0.73\n",
      "> 2022-05-24 18:18:55,329 [info] best iteration=3, used criteria max.validation_accuracy\n",
      "> 2022-05-24 18:18:56,045 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>keras-cifar10-jovyan</td>\n",
       "      <td><div title=\"86337f5c1f06423bb27aa41a2e52ff61\"><a href=\"http://localhost:30060/projects/keras-cifar10-jovyan/jobs/monitor/86337f5c1f06423bb27aa41a2e52ff61/overview\" target=\"_blank\" >...2e52ff61</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 24 15:30:53</td>\n",
       "      <td>completed</td>\n",
       "      <td>cifar10-hp-trainer</td>\n",
       "      <td><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=jovyan</div><div class=\"dictlist\">mlrun/client_version=0.10.0</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=3</div><div class=\"dictlist\">batch_size=64</div><div class=\"dictlist\">epochs=10</div><div class=\"dictlist\">lr=0.0010000000474974513</div><div class=\"dictlist\">training_loss=0.75482177734375</div><div class=\"dictlist\">training_accuracy=0.78131103515625</div><div class=\"dictlist\">validation_loss=0.6332879112170527</div><div class=\"dictlist\">validation_accuracy=0.7853999518738768</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/training_loss.html\">training_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/training_accuracy.html\">training_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/validation_loss.html\">validation_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/validation_accuracy.html\">validation_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/loss_summary.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/accuracy_summary.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/3/lr_values.html\">lr_values.html</div><div title=\"/home/jovyan/data/3\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/iteration_results.csv\">iteration_results</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result225c2d44\" title=\"/files/data/parallel_coordinates.html\">parallel_coordinates</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result225c2d44-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result225c2d44-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result225c2d44\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result225c2d44-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"http://localhost:30060/projects/keras-cifar10-jovyan/jobs/monitor/86337f5c1f06423bb27aa41a2e52ff61/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-05-24 18:19:00,671 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# Run it:\n",
    "grid_params = {\n",
    "    \"batch_size\": [64, 128], \n",
    "    \"lr\": [1e-2, 1e-3],\n",
    "    \"epochs\": [10]\n",
    "}\n",
    "\n",
    "task = mlrun.new_task(\"cifar10-hp-trainer\").with_hyper_params(grid_params,selector=\"max.validation_accuracy\")\n",
    "training_run = training_function.run(task, \n",
    "                                     handler=\"train\",\n",
    "                                     watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "training_function.run(\n",
    "    name=\"cifar10-trainer-training\",\n",
    "    handler=\"train\",\n",
    "    params={\n",
    "        \"batch_size\": 64,\n",
    "        \"lr\": 1e-2,\n",
    "        \"epochs\": 5\n",
    "    },\n",
    "    local=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}