Add literature background convnet

vision/conv_mnist/README.md

@@ -12,6 +12,8 @@ At a high level LeNet (LeNet-5) consists of two parts:
 
 The basic units in each convolutional block are a convolutional layer, a sigmoid activation function, and a subsequent average pooling operation. Each convolutional layer uses a  5×5  kernel and a sigmoid activation function. These layers map spatially arranged inputs to a number of two-dimensional feature maps, typically increasing the number of channels. The first convolutional layer has 6 output channels, while the second has 16. Each  2×2  pooling operation (stride 2) reduces dimensionality by a factor of  4  via spatial downsampling. The convolutional block emits an output with shape given by (batch size, number of channel, height, width).
 
+>**Note:** The original architecture of Lenet5 used the sigmoind activation function. However, this is a a modernized version since it uses the RELU activation function instead.
+
 ## Training
 
 ```shell

vision/conv_mnist/conv_mnist.jl

@@ -1,7 +1,41 @@
-## Classification of MNIST dataset 
-## with the convolutional neural network known as LeNet5.
-## This script also combines various
-## packages from the Julia ecosystem with Flux.
+# # Classification of MNIST dataset using ConvNet
+
+# In this tutorial, we build a convolutional neural network (ConvNet or CNN) known as [LeNet5](https://en.wikipedia.org/wiki/LeNet) 
+# to classify [MNIST](http://yann.lecun.com/exdb/mnist/) handwritten digits.
+
+# LeNet5 is one of the earliest CNNs. It was originally used for recognizing handwritten characters. At a high level LeNet (LeNet-5) consists of two parts: 
+
+# * A convolutional encoder consisting of two convolutional layers.
+# * A dense block consisting of three fully-connected layers.
+
+# The basic units in each convolutional block are a convolutional layer, a sigmoid activation function, 
+# and a subsequent average pooling operation. Each convolutional layer uses a 5×5 kernel and a sigmoid activation function. 
+# These layers map spatially arranged inputs to a number of two-dimensional feature maps, typically increasing the number of channels. 
+# The first convolutional layer has 6 output channels, while the second has 16. 
+# Each 2×2 pooling operation (stride 2) reduces dimensionality by a factor of 4 via spatial downsampling. 
+# The convolutional block emits an output with shape given by (width, height, number of channels,  batch size).
+
+# ![LeNet-5](../conv_mnist/docs/LeNet-5.png)
+
+# Source: https://d2l.ai/chapter_convolutional-neural-networks/lenet.html
+
+# >**Note:** The original architecture of Lenet5 used the sigmoind activation function. However, this is a a modernized version since it uses the RELU activation function instead.
+
+# If you need more information about how CNNs work and related technical concepts, check out the following resources:
+
+# * [Gradient-Based Learning Applied to Document Recognition](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) . This is LeNet5 original paper by Yann LeCunn and others.
+# * [Convolutional Neural Networks for Visual Recognition](https://cs231n.github.io/convolutional-networks/).
+# * [Neural Networks in Flux.jl with Huda Nassar (working with the MNIST dataset)](https://youtu.be/Oxi0Pfmskus).
+# * [Dive into Deep Learning", 2020](https://d2l.ai/chapter_convolutional-neural-networks/lenet.html).
+
+
+# This example demonstrates Flux’s Convolution and pooling layers, the usage of TensorBoardLogger, 
+# how to write out the saved model to the file `mnist_conv.bson`, 
+# and also combines various packages from the Julia ecosystem with Flux. 
+
+
+# To run this example, we need the following packages:
+
 using Flux
 using Flux.Data: DataLoader
 using Flux.Optimise: Optimiser, WeightDecay
@@ -15,9 +49,52 @@ import MLDatasets
 import BSON
 using CUDA
 
-# LeNet5 "constructor". 
-# The model can be adapted to any image size
-# and any number of output classes.
+# We set default values for the arguments for the function `train`:
+
+Base.@kwdef mutable struct Args
+    η = 3e-4             ## learning rate
+    λ = 0                ## L2 regularizer param, implemented as weight decay
+    batchsize = 128      ## batch size
+    epochs = 10          ## number of epochs
+    seed = 0             ## set seed > 0 for reproducibility
+    use_cuda = true      ## if true use cuda (if available)
+    infotime = 1 	     ## report every `infotime` epochs
+    checktime = 5        ## Save the model every `checktime` epochs. Set to 0 for no checkpoints.
+    tblogger = true      ## log training with tensorboard
+    savepath = "runs/"   ## results path
+end
+
+# ## Data
+
+# We create the function `get_data` to load the MNIST train and test data from [MLDatasets](https://github.com/JuliaML/MLDatasets.jl) and reshape them so that they are in the shape that Flux expects. 
+
+function get_data(args)
+    xtrain, ytrain = MLDatasets.MNIST.traindata(Float32)
+    xtest, ytest = MLDatasets.MNIST.testdata(Float32)
+
+    xtrain = reshape(xtrain, 28, 28, 1, :)
+    xtest = reshape(xtest, 28, 28, 1, :)
+
+    ytrain, ytest = onehotbatch(ytrain, 0:9), onehotbatch(ytest, 0:9)
+
+    train_loader = DataLoader((xtrain, ytrain), batchsize=args.batchsize, shuffle=true)
+    test_loader = DataLoader((xtest, ytest),  batchsize=args.batchsize)
+
+    return train_loader, test_loader
+end
+
+# The function `get_data` performs the following tasks:
+
+# * **Loads MNIST dataset:** Loads the train and test set tensors. The shape of the train data is `28x28x60000` and the test data is `28x28x10000`. 
+# * **Reshapes the train and test data:**  Notice that we reshape the data so that we can pass it as arguments for the input layer of the model.
+# * **One-hot encodes the train and test labels:** Creates a batch of one-hot vectors so we can pass the labels of the data as arguments for the loss function. For this example, we use the [logitcrossentropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy) function and it expects data to be one-hot encoded. 
+# * **Creates mini-batches of data:** Creates two DataLoader objects (train and test) that handle data mini-batches of size `128 ` (as defined above). We create these two objects so that we can pass the entire data set through the loss function at once when training our model. Also, it shuffles the data points during each iteration (`shuffle=true`).
+
+# ## Model
+
+# We create the LeNet5 "constructor". It uses Flux's built-in [Convolutional and pooling layers](https://fluxml.ai/Flux.jl/stable/models/layers/#Convolution-and-Pooling-Layers):
+
+
 function LeNet5(; imgsize=(28,28,1), nclasses=10) 
     out_conv_size = (imgsize[1]÷4 - 3, imgsize[2]÷4 - 3, 16)
 
@@ -33,23 +110,23 @@ function LeNet5(; imgsize=(28,28,1), nclasses=10)
           )
 end
 
+
+# **Note:** The model can be adapted to any image size and any number of output classes.
+
 function get_data(args)
     xtrain, ytrain = MLDatasets.MNIST(:train)[:]
     xtest, ytest = MLDatasets.MNIST(:test)[:]
 
-    xtrain = reshape(xtrain, 28, 28, 1, :)
-    xtest = reshape(xtest, 28, 28, 1, :)
 
-    ytrain, ytest = onehotbatch(ytrain, 0:9), onehotbatch(ytest, 0:9)
+# ## Loss function
 
-    train_loader = DataLoader((xtrain, ytrain), batchsize=args.batchsize, shuffle=true)
-    test_loader = DataLoader((xtest, ytest),  batchsize=args.batchsize)
-
-    return train_loader, test_loader
-end
+# We use the function [logitcrossentropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy) to compute the difference between 
+# the predicted and actual values (loss).
 
 loss(ŷ, y) = logitcrossentropy(ŷ, y)
 
+# Also, we create the function `eval_loss_accuracy` to output the loss and the accuracy during training:
+
 function eval_loss_accuracy(loader, model, device)
     l = 0f0
     acc = 0
@@ -64,23 +141,15 @@ function eval_loss_accuracy(loader, model, device)
     return (loss = l/ntot |> round4, acc = acc/ntot*100 |> round4)
 end
 
-## utility functions
+# ## Utility functions
+# We need a couple of functions to obtain the total number of the model's parameters. Also, we create a function to round numbers to four digits.
+
 num_params(model) = sum(length, Flux.params(model)) 
 round4(x) = round(x, digits=4)
 
-# arguments for the `train` function 
-Base.@kwdef mutable struct Args
-    η = 3e-4             # learning rate
-    λ = 0                # L2 regularizer param, implemented as weight decay
-    batchsize = 128      # batch size
-    epochs = 10          # number of epochs
-    seed = 0             # set seed > 0 for reproducibility
-    use_cuda = true      # if true use cuda (if available)
-    infotime = 1 	     # report every `infotime` epochs
-    checktime = 5        # Save the model every `checktime` epochs. Set to 0 for no checkpoints.
-    tblogger = true      # log training with tensorboard
-    savepath = "runs/"    # results path
-end
+# ## Train the model
+
+# Finally, we define the function `train` that calls the functions defined above to train the model.
 
 function train(; kws...)
     args = Args(; kws...)
@@ -106,14 +175,14 @@ function train(; kws...)
     ps = Flux.params(model)  
 
     opt = ADAM(args.η) 
-    if args.λ > 0 # add weight decay, equivalent to L2 regularization
+    if args.λ > 0 ## add weight decay, equivalent to L2 regularization
         opt = Optimiser(WeightDecay(args.λ), opt)
     end
 
     ## LOGGING UTILITIES
     if args.tblogger 
         tblogger = TBLogger(args.savepath, tb_overwrite)
-        set_step_increment!(tblogger, 0) # 0 auto increment since we manually set_step!
+        set_step_increment!(tblogger, 0) ## 0 auto increment since we manually set_step!
         @info "TensorBoard logging at \"$(args.savepath)\""
     end
 
@@ -149,15 +218,31 @@ function train(; kws...)
         if args.checktime > 0 && epoch % args.checktime == 0
             !ispath(args.savepath) && mkpath(args.savepath)
             modelpath = joinpath(args.savepath, "model.bson") 
-            let model = cpu(model) #return model to cpu before serialization
+            let model = cpu(model) ## return model to cpu before serialization
                 BSON.@save modelpath model epoch
             end
             @info "Model saved in \"$(modelpath)\""
         end
     end
 end
 
+# The function `train` performs the following tasks:
+
+# * Checks whether there is a GPU available and uses it for training the model. Otherwise, it uses the CPU.
+# * Loads the MNIST data using the function `get_data`.
+# * Creates the model and uses the [ADAM optimiser](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM) with weight decay.
+# * Loads the [TensorBoardLogger.jl](https://github.com/JuliaLogging/TensorBoardLogger.jl) for logging data to Tensorboard.
+# * Creates the function `report` for computing the loss and accuracy during the training loop. It outputs these values to the TensorBoardLogger.
+# * Runs the training loop using [Flux’s training routine](https://fluxml.ai/Flux.jl/stable/training/training/#Training). For each epoch (step), it executes the following:
+#   * Computes the model’s predictions.
+#   * Computes the loss.
+#   * Updates the model’s parameters.
+#   * Saves the model `model.bson` every `checktime` epochs (defined as argument above.)
+
+# ## Run the example 
+
+# We call the  function `train`:
+
 if abspath(PROGRAM_FILE) == @__FILE__
     train()
 end
-