Final version

README.md

@@ -1 +1,14 @@
-"# phd" 
+# Hyper-parameter optimization in deep learning and transfer learning - Applications to medical imaging
+
+This repo contains my PhD thesis. It was done between 2015 and 2018, and was defended the 15/01/2019 at Télécom ParisTech.
+
+The PhD was done as a collaboration between Télécom ParisTech and Philips Healthcare, my supervisors were Isabelle Bloch, Roberto Ardon and Matthieu Perrot.
+
+Here is the abstract:
+In the last few years, deep learning has changed irrevocably the field of computer vision. Faster, giving better results, and requiring a lower degree of expertise to use than traditional computer vision methods, deep learning has become ubiquitous in every imaging application. This includes medical imaging applications. 
+
+At the beginning of this thesis, there was still a strong lack of tools and understanding of how to build efficient neural networks for specific tasks. Thus this thesis first focused on the topic of hyper-parameter optimization for deep neural networks, i.e. methods for automatically finding efficient neural networks on specific tasks. The thesis includes a comparison of different methods, a performance improvement of one of these methods, Bayesian optimization, and the proposal of a new method of hyper-parameter optimization by combining two existing methods: Bayesian optimization and Hyperband.
+
+From there, we used these methods for medical imaging applications such as the classification of field-of-view in MRI, and the segmentation of the kidney in 3D ultrasound images across two populations of patients. This last task required the development of a new transfer learning method based on the modification of the source network by adding new geometric and intensity transformation layers.
+
+Finally this thesis loops back to older computer vision methods, and we propose a new segmentation algorithm combining template deformation and deep learning. We show how to use a neural network to predict global and local transformations without requiring the ground-truth of these transformations. The method is validated on the task of kidney segmentation in 3D US images. 

latex/chap_hyperopt.tex

@@ -171,7 +171,7 @@ \section{Bayesian Optimization}
 \subsection{Gaussian processes}
 \label{ssec:gp}
 
-A Gaussian process is a supervised learning model mainly used for regression problems. It is a distribution over functions, i.e. from a set of data points, the Gaussian process gives possible functions that fit those points, weighted by their likelihood. The shape and properties of possible functions are defined by a covariance function. When predicting the value of an unseen point, the Gaussian process returns a Normal distribution, with the variance being an estimation of the uncertainty of the model at this point. Predicting multiple points will result in a joint Gaussian distribution. A comprehensive review of the topic can be found in~\textcite{rasmussen2005}
+A Gaussian process is a supervised learning model mainly used for regression problems. It is a distribution over functions, i.e. from a set of data points, the Gaussian process gives possible functions that fit those points, weighted by their likelihood. The shape and properties of possible functions are defined by a covariance function. When predicting the value of an unseen point, the Gaussian process returns a Normal distribution, with the variance being an estimation of the uncertainty of the model at this point. Predicting multiple points will result in a joint Gaussian distribution. A comprehensive review of the topic can be found in~\textcite{rasmussen2005} and from a signal processing point-of-view in~\textcite{perez-cruz2013}.
 
 \subsubsection{Definitions}
 

latex/chap_segmentation.tex

@@ -56,10 +56,12 @@ \subsection{Finding the transformation}
 
 Convolutional neural networks have been used recently to regress the parameters of the registration transformation from the input images (\textcite{miao2016},~\textcite{yang2016}).
 
-Another approach is to estimate a similarity measure from a neural network to be used in an iterative optimization strategy (\textcite{wu2013MICCAI},~\textcite{cheng2015},~\textcite{simonovosky2016MICCAI}).
+Another approach is to estimate a similarity measure from a neural network to be used in an iterative optimization strategy (\textcite{wu2013MICCAI},~\textcite{cheng2015},~\textcite{simonovosky2016MICCAI},~\textcite{Vos2019}).
 
 Recently, method using GANs (\textcite{goodfellow2014}) have been proposed.~\textcite{dalca2018MICCAI} developed a diffeomorphic integration layer coupled with a generative model that allows for registration without the ground-truth deformation fields. In the approach of~\textcite{fan2018MICCAI}, a registration network predicts the deformation and a discrimination network evaluates the quality of the alignment. The networks are trained in the typical GAN fashion. A regularization term is added to the loss of the registration network to make the deformation field smoother.
 
+Instead of a GAN, a variational auto-encoder can be considered to learn a latent space of transformations such as in~\textcite{krebs2018}.
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Segmentation by Implicit Template Deformation}
 \label{sec:implicit}
@@ -130,7 +132,7 @@ \section{Template Deformation via Deep Learning}
 \subsection{Global transformation}
 \label{ssec:seg_global}
 
-A linear geometric transformation for 3D images is represented by a $4 \times 4$ matrix, meaning a total of 16 parameters that can be predicted. Directly predicting those parameters remove control on the kind of transformation that are predicted. In our case, we chose to predict instead translation, rotation and scaling parameters. 
+A linear geometric transformation for 3D images is represented by a $4 \times 4$ matrix, meaning a total of 16 parameters that can be predicted. Directly predicting those parameters, such as in Spatial Transformers (\textcite{jaderberg2015NIPS}), remove control on the kind of transformation that are predicted. In our case, we chose to predict instead translation, rotation and scaling parameters. 
 
 Three parameters are predicted for the translation on each axis, giving the following translation matrix $T$:
 \begin{equation*}
@@ -217,7 +219,7 @@ \subsection{Local deformation}
 \begin{equation}
     L = u(x) + Id = [K_{\sigma} * v](x) + Id
 \end{equation}
-As in the implicit template deformation case, the Gaussian filtering is used here to enforce the regularity of the deformation field.
+As in the implicit template deformation case, the Gaussian filtering is used here to enforce the regularity of the deformation field. It has also been used in the context of registration with deep learning for the same reason (\textcite{krebs2019}).
 
 % We added an $L_2$ penalty term to the deformation fields $\psi$ in the loss function:
 % \begin{equation}

latex/chap_transfer_learning.tex

@@ -248,7 +248,7 @@ \subsection{Intensity Layer}
 \begin{equation*}
 O \left( I_i \right) = K_i \left( I; \theta \right) * I_i + b_i \left( I; \theta \right)
 \end{equation*}
-where $K_i \left( I; \theta \right)$ and $b_i \left( I; \theta \right)$ are the predicted parameters. In the end, two parameters have to be predicted for each feature map. Experiments showed that this layer works best when placed at the lowest level of the U-Net (see Figure~\ref{fig:unet}).
+where $K_i \left( I; \theta \right)$ and $b_i \left( I; \theta \right)$ are the predicted parameters. In the end, two parameters have to be predicted for each feature map. This layer is similar to the Feature-wise Linear Modulation layer from~\textcite{perez2018}. Experiments showed that this layer works best when placed at the lowest level of the U-Net (see Figure~\ref{fig:unet}).
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Results and Discussion}

latex/cover/1ere.tex

@@ -45,7 +45,7 @@
 
 \newcommand{\PhDTitle}{Optimisation d'hyper-paramètres en apprentissage profond et apprentissage par transfert - Applications en imagerie médicale} 	%% Titre de la thèse / Thesis title
 \newcommand{\PhDname}{Hadrien Bertrand} 															%% Civilité, nom et prénom /  Civility, first name and name 
-\newcommand{\NNT}{20XXSACLXXXX} 															%% Numéro National de Thèse (donnée par la bibliothèque à la suite du 1er dépôt)/ National Thesis Number (given by the Library after the first deposit)
+\newcommand{\NNT}{2019SACLT001} 															%% Numéro National de Thèse (donnée par la bibliothèque à la suite du 1er dépôt)/ National Thesis Number (given by the Library after the first deposit)
 
 \newcommand{\ecodoctitle}{Sciences et technologies de l'information et de la communication} 													%% Nom de l'ED. Voir site de l'Université Paris-Saclay / Full name of Doctoral School. See Université Paris-Saclay website
 \newcommand{\ecodocacro}{STIC}																%% Sigle de l'ED. Voir site de l'Université Paris-Saclay / Acronym of the Doctoral School. See Université Paris-Saclay website
@@ -87,35 +87,30 @@
 % Tous les membres du juty listés doivent avoir été présents lors de la soutenance / All the jury members listed here must have been present during the defense.
 
 %%% Membre n°1 (Président) / Member n°1 (President)
-% \newcommand{\jurynameA}{Prénom Nom}
-% \newcommand{\juryadressA}{Statut, Établissement (Unité de recherche)}
-% \newcommand{\juryroleA}{Président}
+\newcommand{\jurynameA}{Laurent Cohen}
+\newcommand{\juryadressA}{Directeur de recherche, Université Paris-Dauphine}
+\newcommand{\juryroleA}{Président}
 
 %%% Membre n°2 (Rapporteur) / Member n°2 (President)
 \newcommand{\jurynameB}{Caroline Petitjean}
 \newcommand{\juryadressB}{Maître de conférence, Université de Rouen}
 \newcommand{\juryroleB}{Rapporteuse}
 
 %%% Membre n°3 (Rapporteur) / Member n°3 (President)
-\newcommand{\jurynameA}{Hervé Delingette}
-\newcommand{\juryadressA}{Directeur de recherche, INRIA Sophia-Antipolis}
-\newcommand{\juryroleA}{Rapporteur}
+\newcommand{\jurynameC}{Hervé Delingette}
+\newcommand{\juryadressC}{Directeur de recherche, INRIA Sophia-Antipolis}
+\newcommand{\juryroleC}{Rapporteur}
 
 %%% Membre n°4 (Examinateur) / Member n°4 (President)
-\newcommand{\jurynameC}{Pierrick Coupé}
-\newcommand{\juryadressC}{Chargé de Recherche, Université de Bordeaux}
-\newcommand{\juryroleC}{Examinateur}
+\newcommand{\jurynameE}{Pierrick Coupé}
+\newcommand{\juryadressE}{Chargé de Recherche, Université de Bordeaux}
+\newcommand{\juryroleE}{Examinateur}
 
 %%% Membre n°4 (Examinateur) / Member n°4 (President)
 \newcommand{\jurynameD}{Jamal Atif}
 \newcommand{\juryadressD}{Professeur, Université Paris-Dauphine}
 \newcommand{\juryroleD}{Examinateur}
 
-%%% Membre n°4 (Examinateur) / Member n°4 (President)
-\newcommand{\jurynameE}{Laurent Cohen}
-\newcommand{\juryadressE}{Directeur de recherche, Université Paris-Dauphine}
-\newcommand{\juryroleE}{Examinateur}
-
 %%% Membre n°5 (Directeur de thèse) / Member n°5 (Thesis supervisor)
 \newcommand{\jurynameF}{Isabelle Bloch}
 \newcommand{\juryadressF}{Professeur, Télécom ParisTech}
@@ -206,7 +201,7 @@
 
 		\small{Spécialité de doctorat: \PhDspeciality} \bigskip %% Spécialité 
 		\vfill  
-		\footnotesize{Thèse présentée et soutenue à \defenseplace, prévue le \defensedate, par} \bigskip
+		\footnotesize{Thèse présentée et soutenue à \defenseplace, le \defensedate, par} \bigskip
 		\vfill
 		\Large{\textbf{\textsc{\PhDname}}} %% Nom du docteur
 		\vfill

latex/main.tex

@@ -20,6 +20,7 @@
 \usepackage{fancyhdr}
 \usepackage{appendix}
 \usepackage[Bjornstrup]{fncychap}
+\usepackage{tikz}
 
 \usepackage{hyperref}
 
@@ -160,6 +161,24 @@
 
 \cleardoublepage
 
+\chapter*{Remerciements}
+
+Je tiens à remercier d'abord mes encadrants Isabelle, Roberto et Matthieu, qui m'ont beaucoup appris pendant ces trois années. C'était un plaisir de travailler avec vous !
+
+Je suis reconnaissant aux membres de mon jury, pour avoir pris le temps de lire et d'évaluer mes travaux de thèse.
+
+A mes chers collègues de Medisys, vous êtes une super équipe, j'ai été heureux de vous connaître. Je remercie en particulier mes collègues doctorants pour les pauses cafés quotidiennes : Alexandre, Gabriel, Ewan, Cecilia, Vincent, Eric, Mariana et Yitian.
+
+J'aimerais aussi remercier tous les membres de l'équipe IMAGES de Télécom, et en particulier Alessio pour m'avoir supporté au bureau et en conférence pendant cette thèse.
+
+A mes amis Emilien, Antoine, Sylvain, Cyril et Florian : \includegraphics[height=1em]{duck.png}
+
+Je remercie Frédéric, pour des conversations semi-régulières et parfois scientifiques tout au long de ma thèse.
+
+Finalement, j'aimerais remercier mes parents et ma soeur, ainsi que le reste de ma famille.
+
+\cleardoublepage
+
 \dominitoc
 {   
     \setstretch{1.1}
@@ -197,10 +216,6 @@
 
 \include{chap_conclusion}
 
-% TODO:
-% - fig 3.9 legend correct ?
-% - fig 3.6 make addition more obvious
-
 \begin{appendices}
     \include{appendix_cholesky}
 \end{appendices}

latex/references.bib

@@ -460,6 +460,23 @@ @article{kamnitsas2017MEDIA
 	year = {2017}
 }
 
+@InProceedings{krebs2018,
+    author = {Krebs, Julian and Mansi, Tommaso and Mailhé, Boris and Ayache, Nicholas and Delingette, Hervé},
+    year = {2018},
+    booktitle = "4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018",
+    title = {Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration},
+    isbn = {978-3-030-00888-8},
+    doi = {10.1007/978-3-030-00889-5_12}
+}
+
+@article{krebs2019,
+    author = {Krebs, Julian and Delingette, Herve and Mailhe, Boris and Ayache, Nicholas and Mansi, Tommaso},
+    year = {2019},
+    title = {Learning a Probabilistic Model for Diffeomorphic Registration},
+    journal = {IEEE Transactions on Medical Imaging},
+    doi = {10.1109/TMI.2019.2897112}
+}
+
 @report{krizhevsky2009,
     title = {Learning Multiple Layers of Features from Tiny Images},
     author = {Alex Krizhevsky},
@@ -644,6 +661,22 @@ @article{pedregosa2011sklearn
  url = "http://scikit-learn.org/stable/index.html"
 }
 
+@article{perez-cruz2013,
+    author={F. Perez-Cruz and S. Van Vaerenbergh and J. J. Murillo-Fuentes and M. Lazaro-Gredilla and I. Santamaria},
+    journal={IEEE Signal Processing Magazine},
+    title={Gaussian Processes for Nonlinear Signal Processing: An Overview of Recent Advances},
+    year={2013},
+    doi={10.1109/MSP.2013.2250352}
+}
+
+@InProceedings{perez2018,
+  title={FiLM: Visual Reasoning with a General Conditioning Layer},
+  author={Ethan Perez and Florian Strub and Harm de Vries and Vincent Dumoulin and Aaron C. Courville},
+  booktitle={AAAI},
+  year={2018},
+  url="https://github.com/ethanjperez/film"
+}
+
 @InProceedings{pham2018ICML,
   title = 	 {Efficient Neural Architecture Search via Parameters Sharing},
   author = 	 {Pham, Hieu and Guan, Melody and Zoph, Barret and Le, Quoc and Dean, Jeff},
@@ -919,6 +952,14 @@ @inproceedings{vincent2008ICML
  doi = {10.1145/1390156.1390294}
 } 
 
+@article{Vos2019,
+  title={A deep learning framework for unsupervised affine and deformable image registration},
+  author={Bob D. de Vos and Floris F. Berendsen and Max A. Viergever and Hessam Sokooti and Marius Staring and Ivana Isgum},
+  journal={Medical image analysis},
+  year={2019},
+  url="https://arxiv.org/abs/1809.06130"
+}
+
 @article{warfield2004,
     author={S. K. Warfield and K. H. Zou and W. M. Wells},
     journal={IEEE Transactions on Medical Imaging},