ee046746_tut_09_3D_Deep_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "\n",
    "#### Dahlia Urbach\n",
    "\n",
    "## Tutorial 09 - Introduction to 3D Deep Learning\n",
    "---\n",
    "<img src=\"./assets/tut_09_teaser0.gif\" style=\"width:800px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://towardsdatascience.com/the-future-of-3d-point-clouds-a-new-perspective-125b35b558b9\">Image source\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://www.youtube.com/watch?v=R-ZXdAEGbiw&feature=youtu.be\">LiDAR point cloud support | Feature Highlight | Unreal Engine\n",
    "</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [Depth Cameras - Quick overview](#)\n",
    "    * Stereo Cameras - Next Week\n",
    "    * [Time of Flight](#-Time-of-Flight-Cameras)\n",
    "* [3D Deep Learning](#-Deep-Learning-on-Point-Clouds)\n",
    "    * [Voxels](#-Voxalization)\n",
    "    * [Multi-View](#-Multi-View-Approach)\n",
    "    * [Point Clouds](#-Apply-Deep-Learning-Directly-on-3D-Point-Clouds)\n",
    "* [3D Applications](#-3D-Deep-Learning-Applications)\n",
    "* [Recommended Tools](#-Recommended-Tools)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_teaser.JPG\" style=\"width:800px\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "\n",
    "# pytorch imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader, Dataset, ConcatDataset\n",
    "# import torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/office/80/000000/depth.png\" style=\"height:50px;display:inline\"> Depth Cameras\n",
    "---\n",
    "* Stereo Cameras - Next week\n",
    "* Time of flight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/android/48/000000/time.png\" style=\"height:50px;display:inline\"> Time of Flight Cameras\n",
    "---\n",
    "* Light travels at approximately a constant speed $c = 3\\times 10^8$ (meters per second).\n",
    "* Measuring the time it takes for light to travel over a distance once can infer distance.\n",
    "* Can be categorized into two types:\n",
    "    1. Direct TOF - switch laser on and off rapidly.\n",
    "    2. Indirect TOF - send out modulated light, then measure phase difference to infer depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### 1. Direct - TOF\n",
    "* **Li**ght **D**etection **A**nd **R**anging (LiDAR) probably best example in computer vision and robotics.\n",
    "* High-energy light pulses limit influence of background illumination.\n",
    "* However, difficulty to generate short light pulses with fast rise and fall times.\n",
    "* High-accuracy time measurement required.\n",
    "* Prone to motion blur.\n",
    "* Sparser as objects grow in distance.\n",
    "<img src=\"./assets/tut_09_LiDAR.GIF\" style=\"width:150px\">\n",
    "<a href=\"https://en.wikipedia.org/wiki/Lidar\">Gif source - Wikipedia</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src=\"./assets/tut_09_sydney.png\" style=\"width:400px\">\n",
    "<a href=\"http://www.acfr.usyd.edu.au/papers/SydneyUrbanObjectsDataset.shtml\">Sydney Dataset</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###### Autonomous Car - LiDAR\n",
    "<img src=\"./assets/tut_09_cam1.JPG\" style=\"width:400px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### SLAM + LIDAT - Zebedee\n",
    "<img src=\"./assets/tut_09_cam2.JPG\" style=\"width:400px\">\n",
    "<a href=\"https://research.csiro.au/robotics/zebedee/\">zebedee</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###### 2. Indirect - TOF\n",
    "* Continuous light waves instead of short light pulses.\n",
    "* Modulation in terms of frequency of sinusoidal waves.\n",
    "* Detected wave after reflection has shifted phase.\n",
    "* Phase shift proportional to distance from reflecting surface.\n",
    "\n",
    "<img src=\"./assets/tut_09_cam3.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_cam4.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/cloud.png\" style=\"height:50px;display:inline\"> Deep Learning on Point Clouds\n",
    "---\n",
    "<img src=\"./assets/tut_o9_pn1.JPG\" style=\"width:800px\">\n",
    "\n",
    "* Calssification\n",
    "* Semantic segmentation\n",
    "* Part segmentation\n",
    "    * Each point belongs to a specific part of the object\n",
    "* ...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "*Qi, Charles R., et al. \"Pointnet: Deep learning on point sets for 3d classification and segmentation.\" CVPR. 2017.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### <img src=\"https://img.icons8.com/bubbles/50/000000/question-mark.png\" style=\"height:50px;display:inline\"> Questions\n",
    "* What are the differences between 2D image an a point cloud?\n",
    "* Why it might be hard to feed a point cloud as neural network input?\n",
    "* What are the benefits of using a point cloud?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/plasticine/100/000000/not-applicable.png\" style=\"height:50px;display:inline\"> Point Clouds Problems\n",
    "---\n",
    "* Point Clouds Vary in Size (not constant)\n",
    "* Unordered Input\n",
    "    * Data is unstructured (no grid)\n",
    "    * Data is invariant to point ordering (permutations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Other Point Clouds Challenges\n",
    "---\n",
    "* Missing data\n",
    "* Noise\n",
    "* Rotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Problem - Point Clouds Vary in Size\n",
    "---\n",
    "<img src=\"./assets/tut_09_pn2.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_pn3.JPG\" style=\"width:800px\">\n",
    "\n",
    "* Different point clouds represent the same object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Problem  - Unordered Input\n",
    "---\n",
    "Point cloud: $N$ **orderless** points, each represented by a $D$ dim vector\n",
    "<img src=\"./assets/tut_09_pn4.JPG\" style=\"width:800px\">\n",
    "How many semi-equal representations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Model needs to be invariant to $N!$ permutations**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  <img src=\"https://img.icons8.com/carbon-copy/100/000000/switch-camera.png\" style=\"height:50px;display:inline\">Alternate 3D Representations - Representations\n",
    "---\n",
    "Solution:\n",
    "* Convert the raw point clouds into Voxels or multiple 2D RGB(D) images\n",
    "\n",
    "<img src=\"./assets/tut_09_pn5.JPG\" style=\"width:800px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Another 3D representation (not in this course):\n",
    "<img src=\"./assets/tut_09_other.JPG\" style=\"width:800px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/carbon-copy/100/000000/sugar-cubes.png\" style=\"height:50px;display:inline\"> Voxalization\n",
    "---\n",
    "Idea: generalize 2D convolutions to regular 3D grids\n",
    "\n",
    "* The straightforward approach: transform the point clouds into a voxel grid by rasterizing and use 3D CNNs\n",
    "<img src=\"./assets/tut_09_vox1.jpg\" style=\"width:300px\">\n",
    "\n",
    "Voxel grid is a 3D grid of equal size volumes (voxels), can be occupied by:\n",
    "* Binary 0/1 - Is there any point within the voxel?\n",
    "* Weighted - The amount of point located within each voxel\n",
    "\n",
    "Usually we use binary occupancy\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "3D convolution uses 4D kernels\n",
    "<img src=\"./assets/tut_09_vox2.JPG\" style=\"width:600px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_voxnet.png\" style=\"width:500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Maturana, Daniel, and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. IROS, 2015."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/plasticine/100/000000/not-applicable.png\" style=\"height:50px;display:inline\"> Voxalization Problems\n",
    "---\n",
    "* Large memory cost\n",
    "* Slow processing time\n",
    "* Limited spatial resolution\n",
    "* Quantization artifacts\n",
    "<img src=\"./assets/tut_09_vox3.JPG\" style=\"width:600px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/multiple-cameras.png\" style=\"height:50px;display:inline\"> Multi-View Approach\n",
    "---\n",
    "Idea: Transfrom the problem into a well known domain (3D$\\rightarrow$2D)\n",
    "\n",
    "* The multi-view approach: project multiple views to 2D and use CNN to process\n",
    "    * How many views do we need? (Another hyper parameter)\n",
    "\n",
    "<img src=\"./assets/tut_09_pn7.png\" style=\"width:600px\">\n",
    "\n",
    "CNN$_1$ - We can use pre-trained networks to extract features followed by fine tune layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multiviewconvolutional neural networks for 3d shape recognition. CVPR, 2015."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/doodle/48/000000/direction-sign.png\" style=\"height:50px;display:inline\"> Apply Deep Learning Directly on 3D Point Clouds\n",
    "---\n",
    "Idea: Most of the raw 3D data are point clouds - Solve the problems!\n",
    "\n",
    "Note: Point Clouds Problems:\n",
    "* Point Clouds Vary in Size (not constant)\n",
    "* Unordered Input\n",
    "    * Data is unstructured (no grid)\n",
    "    * Data is invariant to point ordering (permutations)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/cute-clipart/64/000000/machine-learning.png\" style=\"height:50px;display:inline\"> PointNet\n",
    "---\n",
    "#### Permutation Invariance: Symetric Function\n",
    "$$ f(x_1,x_2,\\dots,x_n) \\equiv f(x_{\\pi_1},x_{\\pi_2},\\dots,x_{\\pi_n}), x_i \\in R^D $$\n",
    "\n",
    "$\\pi$ is a different permutation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Example:\n",
    "$$ f(x_1,x_2,\\dots,x_n) = max\\{x_1,x_2,\\dots,x_n\\}$$\n",
    "$$ f(x_1,x_2,\\dots,x_n) = x_1+x_2+\\dots+x_n$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* How can we construct a family of symmetric functions by neural networks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Observe:\n",
    "$$ f(x_1,x_2,\\dots,x_n) = \\gamma \\circ g(h(x_1),\\dots,h(x_n))$$\n",
    "is symmetric if $g$ is symmetric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_pn8.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/wired/64/000000/diversity.png\" style=\"height:50px;display:inline\"> Basic PointNet Architecture\n",
    "---\n",
    "Empirically, we use multi-layer perceptron (MLP) and max pooling:\n",
    "\n",
    "<img src=\"./assets/tut_09_pn9.JPG\" style=\"width:500px\">\n",
    "\n",
    "Input MLP:\n",
    "$$h(x_i): R^{3} \\rightarrow{} R^{D}$$\n",
    "We can look at it as $D$ functions $\\{h_k\\}_{k=1}^D$ operate on each point where $$h_k(x_i): R^{3} \\rightarrow{} R^{1}$$\n",
    "\n",
    "Pooling layer:\n",
    "$$g(h(x_1),\\dots,h(x_n)): R^{N\\times D} \\rightarrow{} R^{D}$$\n",
    "We apply the pooling over all points for each function $h_k$.\n",
    "$$g(h_k(x_1),\\dots,h_k(x_n)): R^{N\\times 1} \\rightarrow{} R^{1}$$\n",
    "\n",
    "Classification MLP:\n",
    "$$\\gamma \\circ g(h(x_1),\\dots,h(x_n)): R^{D} \\rightarrow{} R^{D_{Num Classes}}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Shared mlp implementation \"trick\":\n",
    "* Use conv layers :  Number of filters $C_{out}$, each filter size is  ${ 1 \\times {C_{in}}} $.\n",
    "* Input: $R^{ N\\times {C_{in}}}$\n",
    "* Output: $R^{N\\times {C_{out}}}$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "MLP:\n",
    "$$h: R^{C_{in}}\\rightarrow{} R^{C_1}\\rightarrow{} \\dots \\rightarrow{} R^{C_{out}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "class Tnet(nn.Module):\n",
    "   def __init__(self, k=3):\n",
    "      super().__init__()\n",
    "      self.k=k\n",
    "      self.conv1 = nn.Conv1d(k,64,1)\n",
    "      self.conv2 = nn.Conv1d(64,128,1)\n",
    "      self.conv3 = nn.Conv1d(128,1024,1)\n",
    "      self.fc1 = nn.Linear(1024,512)\n",
    "      self.fc2 = nn.Linear(512,256)\n",
    "      self.fc3 = nn.Linear(256,k*k)\n",
    "\n",
    "      self.bn1 = nn.BatchNorm1d(64)\n",
    "      self.bn2 = nn.BatchNorm1d(128)\n",
    "      self.bn3 = nn.BatchNorm1d(1024)\n",
    "      self.bn4 = nn.BatchNorm1d(512)\n",
    "      self.bn5 = nn.BatchNorm1d(256)\n",
    "\n",
    "\n",
    "   def forward(self, input):\n",
    "      # input.shape == (bs,n,3)\n",
    "      bs = input.size(0)\n",
    "      xb = F.relu(self.bn1(self.conv1(input)))\n",
    "      xb = F.relu(self.bn2(self.conv2(xb)))\n",
    "      xb = F.relu(self.bn3(self.conv3(xb)))\n",
    "      pool = nn.MaxPool1d(xb.size(-1))(xb)\n",
    "      flat = nn.Flatten(1)(pool)\n",
    "      xb = F.relu(self.bn4(self.fc1(flat)))\n",
    "      xb = F.relu(self.bn5(self.fc2(xb)))\n",
    "\n",
    "      #initialize as identity\n",
    "      init = torch.eye(self.k, requires_grad=True).repeat(bs,1,1)\n",
    "      if xb.is_cuda:\n",
    "        init=init.cuda()\n",
    "      matrix = self.fc3(xb).view(-1,self.k,self.k) + init\n",
    "      return matrix\n",
    "\n",
    "\n",
    "class Transform(nn.Module):\n",
    "   def __init__(self):\n",
    "        super().__init__()\n",
    "        self.input_transform = Tnet(k=3)\n",
    "        self.feature_transform = Tnet(k=64)\n",
    "        self.conv1 = nn.Conv1d(3,64,1)\n",
    "\n",
    "        self.conv2 = nn.Conv1d(64,128,1)\n",
    "        self.conv3 = nn.Conv1d(128,1024,1)\n",
    "\n",
    "\n",
    "        self.bn1 = nn.BatchNorm1d(64)\n",
    "        self.bn2 = nn.BatchNorm1d(128)\n",
    "        self.bn3 = nn.BatchNorm1d(1024)\n",
    "\n",
    "   def forward(self, input):\n",
    "        matrix3x3 = self.input_transform(input)\n",
    "        # batch matrix multiplication\n",
    "        xb = torch.bmm(torch.transpose(input,1,2), matrix3x3).transpose(1,2)\n",
    "\n",
    "        xb = F.relu(self.bn1(self.conv1(xb)))\n",
    "\n",
    "        matrix64x64 = self.feature_transform(xb)\n",
    "        xb = torch.bmm(torch.transpose(xb,1,2), matrix64x64).transpose(1,2)\n",
    "\n",
    "        xb = F.relu(self.bn2(self.conv2(xb)))\n",
    "        xb = self.bn3(self.conv3(xb))\n",
    "        xb = nn.MaxPool1d(xb.size(-1))(xb)\n",
    "        output = nn.Flatten(1)(xb)\n",
    "        return output, matrix3x3, matrix64x64\n",
    "\n",
    "class PointNet(nn.Module):\n",
    "    def __init__(self, classes = 10):\n",
    "        super().__init__()\n",
    "        self.transform = Transform()\n",
    "        self.fc1 = nn.Linear(1024, 512)\n",
    "        self.fc2 = nn.Linear(512, 256)\n",
    "        self.fc3 = nn.Linear(256, classes)\n",
    "\n",
    "\n",
    "        self.bn1 = nn.BatchNorm1d(512)\n",
    "        self.bn2 = nn.BatchNorm1d(256)\n",
    "        self.dropout = nn.Dropout(p=0.3)\n",
    "        self.logsoftmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, input):\n",
    "        xb, matrix3x3, matrix64x64 = self.transform(input)\n",
    "        xb = F.relu(self.bn1(self.fc1(xb)))\n",
    "        xb = F.relu(self.bn2(self.dropout(self.fc2(xb))))\n",
    "        output = self.fc3(xb)\n",
    "        return self.logsoftmax(output), matrix3x3, matrix64x64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://github.com/nikitakaraevv/pointnet/blob/master/nbs/PointNetClass.ipynb\">Code source</a> - Full implementation of PointNet Classification (can be opened in Colab)\n",
    "\n",
    "<a href=\"https://towardsdatascience.com/deep-learning-on-point-clouds-implementing-pointnet-in-google-colab-1fd65cd3a263\">Deep Learning on Point clouds: Implementing PointNet in Google Colab</a> - Nikita Karaev\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### PointNet Classification Network\n",
    "\n",
    "<img src=\"./assets/tut_09_pn10.JPG\" style=\"width:800px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_pn20.gif\" style=\"width:800px\">\n",
    "\n",
    "<a href=\"https://towardsdatascience.com/deep-learning-on-point-clouds-implementing-pointnet-in-google-colab-1fd65cd3a263\">Image source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Transformation Invariance\n",
    "<img src=\"./assets/tut_09_pn21.JPG\" style=\"width:200px\">\n",
    "Learn transformation matrix to improve task performance. We want our network to be invariante to rigid transformation of the object.\n",
    "\n",
    "* Practically, more augmentation over the training dataset also solved the transformation invariance\n",
    "\n",
    "\n",
    "<a href=\"https://medium.com/@luis_gonzales/an-in-depth-look-at-pointnet-111d7efdaa1a\">Image source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Results - Classification\n",
    "<img src=\"./assets/tut_09_pn12.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### PointNet Segmentation Network\n",
    "<img src=\"./assets/tut_09_pn11.JPG\" style=\"width:800px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Extract local features - Describes each point seperatly\n",
    "* Extract global feature - Describes the entire point cloud\n",
    "* Concatenate the local and global features and feed it into a shared mlp - The mlp learns to process the point feature acording to a condition. The condition is described by the global feature vectore.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://github.com/nikitakaraevv/pointnet/blob/master/nbs/PointNetSeg.ipynb\">Code</a> - Full implementation of PointNet Segmentation (can be opened in Colab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Semantic Scene Parsing\n",
    "<img src=\"./assets/tut_09_pn13.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Results - Robustness to Missing Data (Classification example)\n",
    "* Why is PointNet so robust to missing data?\n",
    "<img src=\"./assets/tut_09_pn14.JPG\" style=\"width:500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Visualizing Global  Point Cloud Features\n",
    "<img src=\"./assets/tut_09_pn16.JPG\" style=\"width:800px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Visualize What is Learned by Reconstruction\n",
    "* Salient points are discovered!\n",
    "<img src=\"./assets/tut_09_pn15.png\" style=\"width:800px\">\n",
    "The \"critical points\" are those who influenced the global feature vector, a.k.a the pooling layer. The \"critical\" object's geometry structured is reserved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### Point function visualization\n",
    "For each per-point function $h$ (mlp), calculate the values of $h(p)$ for all the points $p$ in the cube.\n",
    "\n",
    "Random 15 function out of the 1024 learned functions:\n",
    "<img src=\"./assets/tut_09_pn19.JPG\" style=\"width:800px\">\n",
    "\n",
    "* Semi-equivalent to filter response in CNNs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles R., et al. \"PointNet: Deep learning on point sets for 3d classification and segmentation.\" CVPR 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/plasticine/100/000000/not-applicable.png\" style=\"height:50px;display:inline\"> Limitations of PointNet\n",
    "<img src=\"./assets/tut_09_pn17.JPG\" style=\"width:800px\">\n",
    "\n",
    "* No local context for each point\n",
    "* Global feature depends on **absolute** coordinate. Hard to generalize to unseen scene configurations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Points in Metric Space\n",
    "* Learn “kernels” in 3D space and conduct convolution\n",
    "* Kernels have compact spatial support\n",
    "* For convolution, we need to find neighboring points\n",
    "* Possible strategies for range query\n",
    "    * Ball query (results in more stable features)\n",
    "    * k-NN query (faster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/plasticine/100/000000/approve-and-update.png\" style=\"height:50px;display:inline\"> PointNet v2.0: Multi-Scale PointNet\n",
    "\n",
    "<img src=\"./assets/tut_09_pn18.png\" style=\"width:800px\">\n",
    "\n",
    "Repeated layers:\n",
    "* Sample anchor points\n",
    "* Find neighborhood of anchor points\n",
    "* Apply PointNet in each neighborhood to mimic convolution\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi, Charles Ruizhongtai, et al. \"Pointnet++: Deep hierarchical feature learning on point sets in a metric space.\" Advances in neural information processing systems. 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### More Point Clouds DL solutions:\n",
    "* 3DmFV\n",
    "* Dynamic Graph CNN\n",
    "* PCNN\n",
    "* PointCNN\n",
    "* KPConv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/list.png\" style=\"height:50px;display:inline\"> 3D Deep Learning Applications\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Calssification (V)\n",
    "* Semantic segmentation (V)\n",
    "* Part segmentation\n",
    "* Object detection (Upcoming)\n",
    "* Reconstraction\n",
    "* Generation (Upcoming)\n",
    "<!--     * AutoEncoders\n",
    "    * GANs\n",
    "    * Implicit Functions -->\n",
    "* Registration (Upcoming)\n",
    "* Sampling - Downsampling, Upsampling\n",
    "* SLAM\n",
    "* Normal Estimation\n",
    "* and many more..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Registration:\n",
    "Problem statment: Find the rotation and translation transformation between objects\n",
    "\n",
    "<img src=\"./assets/tut_09_pnlk1.JPG\" style=\"width:800px\">\n",
    "\n",
    "* PointNetLK (blue) - Deep Learning, based on Lucas–Kanade method (Tracking lecture)\n",
    "    * Comparing 2 point clouds using PointNet features\n",
    "* ICP (orange) - Classic registration method\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_pnlk2.JPG\" style=\"width:800px\">\n",
    "\n",
    "Both inputs (target and source) are being processed by PointNet architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Aoki, Yasuhiro, et al. \"PointNetLK: Robust & efficient point cloud registration using PointNet.\" CVPR 2019."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Generation\n",
    "Conditional generation\n",
    "<img src=\"./assets/tut_09_pngen1.JPG\" style=\"width:800px\">\n",
    "Free generation\n",
    "<img src=\"./assets/tut_09_pngen2.JPG\" style=\"width:800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### How would you build a point cloud GAN?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### Learning Representations and Generative Models for 3D Point Clouds (Achlioptas et al.)\n",
    "* FC layer as generator\n",
    "* PointNet as discriminator\n",
    "<img src=\"./assets/tut_09_pngen4.png\" style=\"width:300px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_09_pngen3.png\" style=\"width:600px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Achlioptas et al., “Learning Representations and Generative Models for 3D Point Clouds”, ICML 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "More generation methods:\n",
    "* AtlasNet\n",
    "* FoldingNet\n",
    "* PointFlow\n",
    "* OccupancyNetworks\n",
    "* DeepSDF\n",
    "* ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Detection:\n",
    "* Generate object proposals from a view (e.g., using SSD)\n",
    "* Recognize using PointNet\n",
    "<img src=\"./assets/tut_09_pndet1.JPG\" style=\"width:800px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Qi et al., “Frustum PointNets for 3D Object Detection from RGB-D Data”, CVPR 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### <img src=\"https://img.icons8.com/bubbles/50/000000/question-mark.png\" style=\"height:50px;display:inline\"> Questions\n",
    "* What are the differences between 2D image an a point cloud?\n",
    "    * Unstructured\n",
    "    * Vary number of points\n",
    "    * Unordered\n",
    "* Why it might be hard to feed a point cloud as neural network (NN) input?\n",
    "    * Does not rely on a grid\n",
    "    * Does not has a fix size\n",
    "    * Different permutation represent the same point cloud\n",
    "\n",
    "All three diffrences influence directly the abbility of using NN!\n",
    "* What are the benefits of using a point cloud?\n",
    "    * Most sensors raw outputs are point clouds (LiDAR)\n",
    "    * Very efficient representation of 3D data (no empty voxels)\n",
    "    * Reserve geometric details (no quantization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/clouds/100/000000/hand-tools.png\" style=\"height:50px;display:inline\"> Recommended Tools\n",
    "---\n",
    "Python:\n",
    "* Open3D\n",
    "* trimesh\n",
    "* Ipyvolume - Visualization for Notebooks\n",
    "\n",
    "Deep Learning:\n",
    "* Python3D\n",
    "* Kaolin (Pytorch)\n",
    "* TensorFlow Graphics\n",
    "\n",
    "Visualize Tools (drop and view):\n",
    "* CloudCompare\n",
    "* MeshLab\n",
    "\n",
    "For more 3D deep learnig frameworks and datasets:\n",
    "* <a href=\"https://github.com/Yochengliu/awesome-point-cloud-analysis\">awesome-point-cloud-analysis</a>\n",
    "* <a href=\"https://github.com/timzhang642/3D-Machine-Learning#datasets\">3D-Machine-Learning#datasets</a> \n",
    "\n",
    "Datasets:\n",
    "* ModelNet\n",
    "* ShapeNet\n",
    "* PartNet\n",
    "* Sydney Urban Opject DAtaset\n",
    "* Stanford 3D\n",
    "* KITTI\n",
    "* ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* 3D Deep Learning\n",
    "    * General (Both highly recomanded):\n",
    "        *  <a href=\"https://www.youtube.com/watch?time_continue=6&v=vfL6uJYFrp4&feature=emb_logo\">3D Deep Learning Tutorial from SU lab at UCSD</a> - Hao Su\n",
    "        *  <a href=\"https://www.youtube.com/watch?v=wLU4YsC_4NY\n",
    "o\">Geometric deep learning</a> - Micahel Bronstein\n",
    "    * <a href=\"https://www.youtube.com/watch?v=Cge-hot0Oc0&t=24s\">PointNet</a> \n",
    "    * <a href=\"https://www.youtube.com/watch?v=HIUGOKSLTcE\">3DmFV</a>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "----\n",
    "* Slides - <a href=\"http://www.itzikbs.com/category/research-blog\">Yizhak (Itzik) Ben-Shabat</a>,  <a href=\"https://ci2cv.net/people/simon-lucey/\">Simon Lucey (CMU)</a>, <a href=\"https://cseweb.ucsd.edu/~haosu/\">Hao Su, Jiayuan Gu and Minghua Liu(UCSanDiego) </a>\n",
    "* Multiple View Geometry in Computer Vision - Hartley and Zisserman - Sections 9,10\n",
    "* <a href=\"https://www.springer.com/gp/book/9781848829343\">Computer Vision: Algorithms and Applications</a> - Richard Szeliski - Sections 11,12\n",
    "\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}