ee046746_tut_14_deep_computational_imaging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "\n",
    "#### Elias Nehme\n",
    "\n",
    "## Tutorial 14 - Deep Computational Imaging\n",
    "---\n",
    "<img src=\"./assets/tut_14_teaser.gif\" style=\"width:800px\">\n",
    "\n",
    "* <a href=\"https://www.nature.com/articles/s41377-020-00403-7\">Image source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [What is Computational Imaging?](#-What-is-Computational-Imaging?)\n",
    "* [Compressive Imaging](#-Compressive-Imaging)\n",
    "    * [Depth Encoding PSF](#-Depth-Encoding-PSF)\n",
    "* [Deep \"Optics\"](#-Deep-Optics)\n",
    "    * [Computer Vision Pipelines](#-Computer-Vision-Pipelines)\n",
    "    * [Differentiable Optics](#-Differentiable-Optics)\n",
    "* [Applications](#-Applications)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/64/000000/fantasy.png\" style=\"height:50px;display:inline\"> What is Computational Imaging?\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_ci_1.jpg\" width=\"800\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/64/000000/fantasy.png\" style=\"height:50px;display:inline\"> What is Computational Imaging?\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_ci_2.jpg\" width=\"800\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Compressive Imaging\n",
    "--- \n",
    "* Depth encoding PSF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Measurement is a 2D image:\n",
    "\n",
    "<img src=\"./assets/tut14_cs_0.jpg\" width=\"400\">\n",
    "\n",
    "* <a href=\"https://www.osapublishing.org/optica/fulltext.cfm?uri=optica-5-1-1&id=380297\">Image source - Optica 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Recovery is a 3D volume:\n",
    "\n",
    "<img src=\"./assets/tut14_cs_1.gif\" width=\"400\">\n",
    "\n",
    "* What?? How?!\n",
    "* <a href=\"https://www.osapublishing.org/optica/fulltext.cfm?uri=optica-5-1-1&id=380297\">Image source - Optica 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Depth Encoding Impulse Response/Point Spread Function (PSF)\n",
    "    * Main idea is to encode depth in the shape generated on the 2D sensor\n",
    "\n",
    "<img src=\"./assets/tut14_cs_2.gif\" width=\"600\">\n",
    "\n",
    "* <a href=\"https://www.osapublishing.org/optica/fulltext.cfm?uri=optica-5-1-1&id=380297\">Image source - Optica 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Writing down the problem in matrix formulation\n",
    "\n",
    "<img src=\"./assets/tut14_cs_3.jpg\" width=\"800\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Solution given by \"MAP\" estimator under certain conditions\n",
    "\n",
    "<img src=\"./assets/tut14_cs_4.jpg\" width=\"800\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/zip.png\" style=\"height:50px;display:inline\"> Depth Encoding PSF\n",
    "--- \n",
    "\n",
    "* Overall framework:\n",
    "\n",
    "<img src=\"./assets/tut14_cs_5.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://www.osapublishing.org/optica/fulltext.cfm?uri=optica-5-1-1&id=380297\">Image source - Optica 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/wired/64/000000/switch-camera.png\" style=\"height:50px;display:inline\"> Deep Optics\n",
    "--- \n",
    "* Computer Vision Pipelines\n",
    "* Differentiable Optics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_1.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_2.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_3.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_4.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_5.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_6.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_7.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_8.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_9.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "* Animal vision is adapted to the surrounding and the day-to-day \"task\"\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_10.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "* \"Standard\" deep image processing\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_11.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "* Deep computational imaging\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_12.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "* Image classification with specialized \"optics\"\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_13.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "* Main idea: optimize the optics and the algorithm jointly to excel in the final task\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_14.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-identification.png\" style=\"height:50px;display:inline\"> Computer Vision Pipelines\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_cv_pipe_15.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_1.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_2.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_3.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_4.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_5.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_6.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_7.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_8.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_9.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_10.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_11.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_12.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_13.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_14.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/camera-addon.png\" style=\"height:50px;display:inline\"> Differentiable Optics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut14_diff_opt_15.jpg\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/wired/64/000000/untested.png\" style=\"height:50px;display:inline\"> Applications\n",
    "--- \n",
    "* Extended Depth of Field\n",
    "* Monocular Depth Estimation / Depth from Defocus\n",
    "* High Dynamic Range Imaging\n",
    "* Video Compressive Sensing\n",
    "* Computational Microscopy (Will not show examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/aperture.png\" style=\"height:50px;display:inline\"> Application 1: Extended Depth of Field (EDOF)\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_edof_0.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://dl.acm.org/doi/10.1145/3197517.3201333\">Image source - ACM SIGGRAPH 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/aperture.png\" style=\"height:50px;display:inline\"> Application 1: Extended Depth of Field (EDOF)\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_edof_1.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://dl.acm.org/doi/10.1145/3197517.3201333\">Image source - ACM SIGGRAPH 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/aperture.png\" style=\"height:50px;display:inline\"> Application 1: Extended Depth of Field (EDOF)\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_edof_2.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://dl.acm.org/doi/10.1145/3197517.3201333\">Image source - ACM SIGGRAPH 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/ios-glyphs/64/000000/abscissa.png\" style=\"height:50px;display:inline\"> Application 2: Monocular Depth Estimation\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_dfd_0.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_ICCV_2019/papers/Chang_Deep_Optics_for_Monocular_Depth_Estimation_and_3D_Object_Detection_ICCV_2019_paper.pdf\">Image source - ICCV 2019</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/ios-glyphs/64/000000/abscissa.png\" style=\"height:50px;display:inline\"> Application 2: Monocular Depth Estimation\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_dfd_1.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_ICCV_2019/papers/Chang_Deep_Optics_for_Monocular_Depth_Estimation_and_3D_Object_Detection_ICCV_2019_paper.pdf\">Image source - ICCV 2019</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./assets/input_vid_dfd.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Video\n",
    "Video(\"./assets/input_vid_dfd.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./assets/output_vid_dfd.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(\"./assets/output_vid_dfd.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./assets/bb_3d.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(\"./assets/bb_3d.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/officel/60/000000/width.png\" style=\"height:50px;display:inline\"> Application 3: High Dynamic Range Imaging\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_hdr_0.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_CVPR_2020/papers/Metzler_Deep_Optics_for_Single-Shot_High-Dynamic-Range_Imaging_CVPR_2020_paper.pdf\">Image source - CVPR 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/officel/60/000000/width.png\" style=\"height:50px;display:inline\"> Application 3: High Dynamic Range Imaging\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_hdr_1.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_CVPR_2020/papers/Metzler_Deep_Optics_for_Single-Shot_High-Dynamic-Range_Imaging_CVPR_2020_paper.pdf\">Image source - CVPR 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/officel/60/000000/width.png\" style=\"height:50px;display:inline\"> Application 3: High Dynamic Range Imaging\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_hdr_2.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_CVPR_2020/papers/Metzler_Deep_Optics_for_Single-Shot_High-Dynamic-Range_Imaging_CVPR_2020_paper.pdf\">Image source - CVPR 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/officel/60/000000/width.png\" style=\"height:50px;display:inline\"> Application 3: High Dynamic Range Imaging\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_hdr_3.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://openaccess.thecvf.com/content_CVPR_2020/papers/Metzler_Deep_Optics_for_Single-Shot_High-Dynamic-Range_Imaging_CVPR_2020_paper.pdf\">Image source - CVPR 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/material-outlined/48/000000/video-trimming.png\" style=\"height:50px;display:inline\"> Application 4: Video Compressive Sensing\n",
    "--- \n",
    "\n",
    "* Challenge is to learn the binary masks $\\Phi$ to recover video $x$ from snapshot $y$\n",
    "\n",
    "<img src=\"./assets/tut14_app_vid_cs_0.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://www.sciencedirect.com/science/article/pii/S1051200419301459\">Image source - DSP Magazine 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/material-outlined/48/000000/video-trimming.png\" style=\"height:50px;display:inline\"> Application 4: Video Compressive Sensing\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_vid_cs_1.jpg\" width=\"800\">\n",
    "\n",
    "* <a href=\"https://ieeexplore.ieee.org/document/9064896\">Image source - ICCP 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/material-outlined/48/000000/video-trimming.png\" style=\"height:50px;display:inline\"> Application 4: Video Compressive Sensing\n",
    "--- \n",
    "\n",
    "<img src=\"./assets/tut14_app_vid_cs_2.jpg\" width=\"400\">\n",
    "\n",
    "* <a href=\"https://ieeexplore.ieee.org/document/9064896\">Image source - ICCP 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/material-outlined/48/000000/video-trimming.png\" style=\"height:50px;display:inline\"> Application 4: Video Compressive Sensing\n",
    "--- \n",
    "\n",
    "* 64 frames recovered from 4 measurements (16 frames/capture)\n",
    "\n",
    "<img src=\"./assets/tut14_app_vid_cs_3.gif\" width=\"400\">\n",
    "\n",
    "* <a href=\"https://ieeexplore.ieee.org/document/9064896\">Image source - ICCP 2020</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> Available resources online\n",
    "---\n",
    "* https://github.com/computational-imaging/opticalCNN\n",
    "* https://github.com/vsitzmann/deepoptics\n",
    "* https://github.com/computational-imaging/DeepOpticsHDR\n",
    "* https://github.com/EliasNehme/DeepSTORM3D\n",
    "* https://github.com/computational-imaging/DepthFromDefocusWithLearnedOptics\n",
    "* etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* End-to-end Optimization of Optics and Image Processing - <a href=\"https://www.youtube.com/watch?v=iJdsxXOfqvw&t=266s&ab_channel=DingzeyuLi\">Vincent Sitzmann</a>\n",
    "* Neural Sensors - <a href=\"https://www.youtube.com/watch?v=tTYHoxA2RVg&ab_channel=StanfordComputationalImagingLab\">J.N.P Martel</a>\n",
    "* High Dynamic Range Imaging - <a href=\"https://www.youtube.com/watch?v=Pla8p9Nqlb8&ab_channel=StanfordComputationalImagingLab\">Christopher Metzler</a> \n",
    "* 3D Single Molecule Localization Microscopy - <a href=\"https://app.quantitativebioimaging.com/video/17\">Elias Nehme</a>  \n",
    "* Towards Neural Imaging & Signal Processing - <a href=\"https://www.youtube.com/watch?v=vTio0tuizHw&t=3984s&ab_channel=IEEESignalProcessingSociety\">Gordon Wetzstein</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "----\n",
    "* <a href=\"https://vsitzmann.github.io/deepoptics/\">End-to-end optimization of optics and image processing</a> - Vincent Sitzmann\n",
    "* <a href=\"https://dl.acm.org/doi/abs/10.1145/3388769.3407486\">ACM SIGGRAPH 2020 Courses</a> - Yifan (Evan) Peng, Ashok Veeraraghavan, Wolfgang Heidrich, Gordon Wetzstein\n",
    "* <a href=\"http://stanford.edu/class/ee367/\">Stanford EE367/CS448I </a> (Computational Imaging and Display) - Gordon Wetzstein\n",
    "* <a href=\"https://sites.google.com/view/sps-space/home?authuser=0\">IEEE SPACE Webinar</a> - IEEE Computational Imaging TC\n",
    "* Research papers:\n",
    "    * <a href=\"https://dl.acm.org/doi/10.1145/3197517.3201333\">End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging</a>\n",
    "    * <a href=\"https://openaccess.thecvf.com/content_ICCV_2019/papers/Chang_Deep_Optics_for_Monocular_Depth_Estimation_and_3D_Object_Detection_ICCV_2019_paper.pdf\">Deep Optics for Monocular Depth Estimation and 3D Object Detection</a>\n",
    "    * <a href=\"https://openaccess.thecvf.com/content_CVPR_2020/papers/Metzler_Deep_Optics_for_Single-Shot_High-Dynamic-Range_Imaging_CVPR_2020_paper.pdf\">Deep Optics for Single-shot High-dynamic-range Imaging</a>\n",
    "    * <a href=\"https://ieeexplore.ieee.org/document/9064896\">Neural Sensors: Learning Pixel Exposures for HDR Imaging and Video Compressive Sensing With Programmable Sensors</a>\n",
    "    * <a href=\"https://arxiv.org/abs/1709.07223\">Convolutional neural networks that teach microscopes how to image</a>\n",
    "    * <a href=\"https://www.nature.com/articles/s41592-020-0853-5\">DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning</a>\n",
    "    * <a href=\"http://www.computationalimaging.org/publications/deepopticsdfd/\">Depth From Defocus With Learned Optics</a>\n",
    "    * etc.\n",
    "\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}