Suffering

latex/chap_conclusion.tex

@@ -7,7 +7,9 @@ \chapter{Conclusion}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Summary of the contributions}
 
-When work on this thesis began in early 2016, deep learning had already shown its worth on natural images, but contributions in medical imaging were rare. There was still a strong lack of tools and understanding on how to build an architecture adapted to a problem, which naturally led us to the topic of hyper-parameter optimization. We resume the work we did in this area below. Now armed with a set of tools to quickly find good architectures, the second part of this thesis focused on applications. Questions of transfer learning and template deformation, and their link with deep learning were explored in this context.
+When work on this thesis began in early 2016, deep learning had already shown its worth on natural images, but contributions in medical imaging were rare. There was still a strong lack of tools and understanding on how to build architectures adapted to a specific problem, which naturally led us to the topic of hyper-parameter optimization, in order to avoid tedious architecture handcrafting and hyper-parameter fine tuning. 
+
+Now armed with a set of tools to quickly find good models, the second part of this thesis focused on applications. Questions of transfer learning and template deformation, and their link with deep learning were explored in this context. They came from the lack of data so common in medical imaging, and thus a need to re-use the limited knowledge at our disposal as much as possible. 
 
 \paragraph*{Hyper-parameter optimization}
 \begin{itemize}
@@ -28,17 +30,23 @@ \section{Summary of the contributions}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Future Work}
 
-Even though all the contributions listed in the previous section have immediate ways they could be improved, those are discussed in their respective chapters. Here we take a longer term perspective and ask ourselves what we would do with more time, taking into account how we believe the field to evolve.
+Even though all the contributions listed in the previous section have immediate ways they could be improved, those are discussed in their respective chapters. Here we take a longer term perspective and ask ourselves what we would do with more time, taking into account how we believe the field is going to evolve.
 
 \paragraph*{User interactions.}
 An important aspect of the implicit template deformation framework is that it can integrate user interactions. After the template is deformed, a user can add or remove points from the segmentation. Those are taken into account by the method, which produces a new refined segmentation. This process can be repeated as many times as required. This kind of user interaction is key in medical practice but deep learning provides no easy way to incorporate them. While some attempts have been made such as~\textcite{cicek2016MICCAI} which provides a partial segmentation as an additional input of the segmentation network, none of them forces the network to make use of them. In our method of template deformation, user input could be incorporated as additional constraints on the deformation field, while forcing the final segmentation to still be an acceptable shape.
 
 \paragraph*{A shift in applications.}
 Deep learning has fulfilled one of its promises: many of the tasks that were once difficult are now easily solvable with deep learning. The classification task of MRI field-of-view we worked on is a good example. Once the dataset is ready, automated methods of hyper-parameter optimization can be used to find a very efficient model with little manual effort. As automation tools improves and becomes more accessible\footnote{See for example \href{https://cloud.google.com/automl/}{Google AutoML}.}, many tasks will stop requiring image processing expertise to be solved. The focus will move to more challenging tasks, where the difficulty can come from a lack of data (rare disease, high variability), greater complexity such as segmentation of small structures (due to the reliance of neural networks on pooling layers), registration, surgical planning, ...
 
+\paragraph*{An integration of older methods.}
+This thesis first started completely focused on deep leaning and how it could be used to solve medical imaging problems. By the end of it, we had started working on a hybrid approach of deep learning and template deformation. Pre-deep learning methods were tailored for medical tasks, and were mostly discarded with the success of deep learning. But deep learning was developed first for natural images applications, without accounting for the differences between natural and medical images. The ideas used in those older methods (such as the use of shape information) are still relevant, and we expect their integration with deep learning will move the field forward in coming years.
+
 \paragraph*{Multi-modality, multi-task learning.}
 The idea of transfer learning is to share knowledge between more or less loosely connected problems. Expanding on this, we could imagine a multi-input multi-output model that shares knowledge over many tasks and modalities at the same time. A first step in this direction could be the construction of modality-specific pre-trained models, a medical imaging equivalent to ImageNet pre-trained models. This requires that huge dataset are available per modality, but efforts in this direction are ongoing. For example, the NIH recently released an \href{https://nihcc.app.box.com/v/ChestXray-NIHCC}{X-Ray dataset} of over $100,000$ images that could be used to build a standard X-Ray pre-trained model. If ImageNet pre-trained models can improve performance on most medical tasks, it seems sensible a more specific model would improve performance even more.
 
+\paragraph*{Differential privacy.}
+Due to the sensitive nature of medical images, sharing between research institutes or hospitals is difficult. A solution to this problem could be through the use of differential privacy. It is a mathematical framework in which the output of an algorithm is mostly the same, whether the data from one particular patient is included or not. This prevents an opponent from recovering patient information from the algorithm (it is possible to reconstruct training set images from computer vision systems, including deep neural networks, see for example~\textcite{fredrikson2015}). There has been some work to make deep neural networks differentially private (\textcite{abadi2016}), and this would allow institutions to release models trained on private data with strong guarantees of patient privacy.
+
 
 % All of the contributions presented can be developed further as we discuss in this section.
 

latex/chap_introduction.tex

@@ -20,8 +20,6 @@ \chapter{Introduction}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Context}
 
-This thesis is at the cross-road of medical image analysis and deep learning, and explores how the recent advances in deep learning can be leveraged to improve the performance on tasks where traditional computer vision methods were insufficient. 
-
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{Medical Imaging}
 
@@ -33,46 +31,40 @@ \subsection{Medical Imaging}
 
 While these may seem to make the problem simpler at first glance, challenges in medical image analysis come in two categories. The first challenge is variability, either intra-subject (changes in the body with time, or during image acquisition as the patient breathes and moves) or inter-subject (in the shape, size and location of bones and organs). Variability also comes from external sources: patterns of noise specific to a machine, image resolution and contrasts depending on the image protocol, field-of-view depending on how the clinicians handle the probe ... 
 
-The second challenge comes from the difficulty of acquiring data and therefore the low amount of data available. This difficulty is caused by the time and expertise required for the acquisition itself, the sensitivity of the data that adds ethical, administrative and bureaucratic overhead, and sometimes the rarity of a disease.\footnote{Initiatives are underway to create big and accessible medical images databases. (See for examples the \href{https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_repositories.html}{NIH Data Sharing Repositories} or the \href{https://www.kaggle.com/datasets?tagids=4202}{Kaggle Healthcare datasets})}
+The second challenge comes from the difficulty of acquiring data and therefore the low amount of data available. This difficulty comes in many forms: the acquisition of images requires time and expertise, the sensitivity of the data adds ethical, administrative and bureaucratic overhead (the recent GDPR laws come to mind), and sometimes the rarity of a disease makes it simply impossible to acquire large amounts of data.\footnote{Initiatives are underway to create big and accessible medical images databases. See for examples the \href{https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_repositories.html}{NIH Data Sharing Repositories} or the \href{https://www.kaggle.com/datasets?tagids=4202}{Kaggle Healthcare datasets}.} 
+
+And acquiring the data is not enough! The data needs to be annotated, a task that often needs to be done by a clinical expert. Image segmentation is an important problem, but the manual creation of the necessary ground-truth segmentations is a time consuming activity that puts a strong limit on the size of medical images databases. A rough estimation: at an average of 5 minutes per image (an optimistic time in many cases, in particular for 3D images), creating the ground-truth segmentations of a 100 images requires slightly over 8 hours of clinician time.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{Deep Learning}
 
-These last few years, it has been impossible to talk about medical image analysis without talking about deep learning. The inescapable transformation brought with deep learning was made possible with the advent of cheap memory storage and computing power. %It is not, however, the first time artificial neural networks where applied to medical imaging problems.  
+These last few years, it has been impossible to talk about medical image analysis without talking about deep learning. The inescapable transformation brought with deep learning was made possible with the advent of cheap memory storage and computing power. At first glance, deep learning gives much better results than traditional computer vision algorithms, while often being faster. From its first big success in the 2012 ImageNet competition won by~\textcite{krizhevsky2012NIPS}, deep learning has had a string of successes that now make it ubiquitous in computer vision, including medical imaging.
 
-This change brought with it its own set of challenges. The huge amount of resources invested into the field results in a deluge of publications where it is difficult to keep up-to-date and separate the noise from the actual progress.
+This change brought with it its own set of challenges. The huge amount of resources invested into the field results in a deluge of publications where it is difficult to keep up-to-date and separate the noise from the actual progress. New algorithms and technologies go from research to production to widely used so fast, the field has completely changed in the duration of this thesis.
 
 To give some perspectives on the progress, at the start of this thesis in early 2016: 
 \begin{itemize}
     \item Tensorflow (\textcite{tensorflow2015}) was just released and a nightmare to install - now it works everywhere from desktop to smartphone and has been cited in over 5000 papers.
     \item The original GAN paper by Goodfellow \textit{et al} was published mid 2014 (\textcite{goodfellow2014}). Early 2016, GANs had the reputation of being incredibly difficult to train and unusable outside of toy datasets. Three years and 5000 citations later, GANs have been applied to a wide range of domains, including medical imaging.
-    \item The omnipresent U-Net architecture had just made its debut a few months earlier at MICCAI 2015 (\textcite{ronneberger2015MICCAI}).
+    \item The now omnipresent U-Net architecture had just made its debut a few months earlier at MICCAI 2015 (\textcite{ronneberger2015MICCAI}).
 \end{itemize}
 
-In the context of medical imaging, deep learning also brings technical challenges. Two of the most common criticisms are the lack of interpretability of neural networks and the lack of robustness. They are barriers to the adoption of deep learning in clinical use and resolving those issues would open the doors to new tasks such as diagnosis or surgical intervention. 
+%In the context of medical imaging, deep learning also brings technical challenges. Two of the most common criticisms are the lack of interpretability of neural networks and the lack of robustness. They are barriers to the adoption of deep learning in clinical use and resolving those issues would open the doors to new tasks such as diagnosis or surgical intervention. 
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\section{Contributions}
+\section{Contributions and Outline}
 
-During this thesis we explored a range of problems, from theoretical to applied. This work contains the following contributions (listed in the order they appear in the thesis):
-\begin{itemize}
-    \item A performance improvement of Bayesian optimization by using an incremental Cholesky decomposition. Chapter~\ref{chap:hyperopt}, Section~\ref{sec:cholesky}.
-    \item A theoretical bound on the performance of random search. Chapter~\ref{chap:hyperopt}, Section~\ref{sec:compare}.
-    \item A new hyper-parameter optimization method combining Hyperband and Bayesian optimization. Chapter~\ref{chap:hyperopt}, Section~\ref{sec:cap}. Published as~\textcite{bertrand2017CAp}.
-    \item A method to solve a classification problem of MRI field-of-view. Chapter~\ref{chap:hyperopt}, Section~\ref{sec:isbi}. Published as~\textcite{bertrand2017ISBI}.
-    \item A new transfer learning method and its application to the segmentation of the kidney in 3D ultrasound images. Chapter~\ref{chap:transfer}. Filled as a patent, under review.
-    \item A statistical shape model approach using deep learning. Chapter~\ref{chap:seg}.
-\end{itemize}
+This thesis is at the cross-road of medical image analysis and deep learning. It first started as an exploration of how to use deep learning on medical imaging problems. The first roadblock encountered was the construction of neural networks specific to a problem. Their was a lack of understanding of the effect of each component of the network on the task to be solved (this is still the case). How many convolutional layers are needed to solve this task ? Is it better to have more filters or bigger filters ? What is the best batch size ? The lack of answers to those questions led us to the topic of hyper-parameter optimization; if we cannot lean on theoretical foundations to build our models, then at least we can automate the search of the best model for a given task and have an empirical answer.
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\section{Outline}
+Once equipped with hyper-parameter optimization tools, we turned to applications, first the classification of field-of-view in MR images, then the segmentation of the kidney in 3D ultrasound. This last problem was examined in a transfer learning setting and led us to the development of a new transfer learning method. 
 
-This thesis is structured in three parts, each starting with a review of the relevant literature.
+In the final part of this thesis we returned to older computer vision methods, notably template deformation methods, and proposed a new method to combine them with deep learning.
 
-Chapter~\ref{chap:hyperopt} discusses the topic of hyper-parameter optimization in the context of deep learning. We focus on three methods: random search, Bayesian optimization and Hyperband, presenting a performance comparison, an improvement and a method combining them. The chapter ends with an application to the problem of MRI field-of-view classification.
+This thesis is structured in three parts, each starting with a review of the relevant literature. 
 
-Chapter~\ref{chap:transfer} introduces a new transfer learning methods in order to solve the task of segmenting the kidney in 3D ultrasound across two populations: healthy adults and sick children.
-
-Chapter~\ref{chap:seg} also deals with to the segmentation of the kidney, but using a statistical shape model and a neural network predicting a deformation field for the model.
-
-Chapter~\ref{chap:conclusion} summarizes our conclusions and discusses possible future works.
+\begin{itemize}
+    \item \textbf{Chapter~\ref{chap:hyperopt}} discusses the topic of hyper-parameter optimization in the context of deep learning. We focus on three methods: random search, Bayesian optimization and Hyperband. The chapter includes (1) a performance improvement of Bayesian optimization by using an incremental Cholesky decomposition; (2) a theoretical bound on the performance of random search and a comparison of random search and Bayesian optimization in a practical setting; (3) a new hyper-parameter optimization method combining Hyperband and Bayesian optimization, published as~\textcite{bertrand2017CAp}; (4) an application of Bayesian optimization to solve a classification problem of MRI field-of-view, published as~\textcite{bertrand2017ISBI}.
+    \item \textbf{Chapter~\ref{chap:transfer}} introduces a new transfer learning method in order to solve the task of segmenting the kidney in 3D ultrasound across two populations: healthy adults and sick children. The challenge comes from the high variability of the children images and the low amount of images available, making transfer learning methods such as fine-tuning insufficient. Our method modifies the source network by adding layers to predict geometric and intensity transformations that are applied to the input image. The method was filled as a patent and is currently under review.
+    \item \textbf{Chapter~\ref{chap:seg}} presents a segmentation method that combines deep learning with template deformation. Building on top of the \textit{implicit template deformation} framework, a neural network is used to predict the parameters of a global and a local transformations, which are applied to a template to segment a target. The method requires only pairs of image and ground-truth segmentation to work, the ground-truth transformations are not required. The method is tested on the task of kidney segmentation in 3D US.
+    \item \textbf{Chapter~\ref{chap:conclusion}} summarizes our conclusions and discusses possible future works.
+\end{itemize}