Sopt-v0.1-beta.1

Updated the optimization code to use Tensorflow 2.2 (from Tensorflow 1.x). The code currently supports:

• the Generalized Gauss-Newton optimization methods: 
  • the Levenberg-Marquardt algorithm the Curveball algorithm.
  • a minibatch LM-type algorithm- the nonlinear conjugate gradient algorithm (PR)
• backtracking line searches.- linear conjugate gradient algorithm 
• an interface to the scipy.optimize methods (CG, BFGS, L-BFGS-B, Newton-CG, and trust-ncg algorithms).

All the algorithms rely on the reverse-mode AD for gradient and higher order derivative calculations. These calculations are all matrix-free and rely on matrix-vector-products. They can be used for GPU-based optimization. 

For the second order methods, since matrix vector products involving the Generalized Gauss-Newton matrix are much more efficient than those involving the full Hessian matrix, I have only actively implemented and tested the GGN-based algorithms: the LM algorithm and the Curveball algorithm.

However, if we want to use the full Hessian matrix for optimization, we can still do that by interfacing with the Newton-CG and trust-NCG methods available in the Scipy Optimize package. In these cases, the objective value, gradient, and the matrix-vector product would be calculated within Tensorflow (e.g. in the GPU), then transferred to the Scipy (in the CPU).