PyTorch auto-grad mechanism implemented in Cpp (only the essential parts). More modules and functions could be easily added.
Dependency Chain of Cpp Modules:
autograd <- optims <- modules <- blocks <- funcs <- nn <- parameters
Matrix & Tensor operations: nn.h
Matrix: basic operations (transpose, multiplication, slice, etc.) and also neural network specific operations (softmax, normalize, position encoding, etc.) Tensor: built on matrix; supports gradient accumulation and back-propagation (see differentiate below)
Functions: funcs.h
Includes matrix multiplication, matrix add a matrix (or a vector), matrix point-wise multiplication, sum of matrix elements, matrix scaling, point-wise Tanh and MSE loss. Each of them should have two implemented member functions: apply (apply the function to arguments), and differentiate (define how the gradient is calculated given the gradient chain delivered to it)
Blocks: blocks.h
Includes linear layer implementation. More blocks (like convolution block) could be added if needed.
Modules: modules.h
Includes 3-layer MLP implementation. More modules (like CNN) could be added if needed.
Optimizers: optims.h
The implementation of optimizer is on another computer but I have no access to it currently. This part is cleared and left blank temporarily. You are welcomed to contribute code!
https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html