Papers

• Understanding the difficulty of training deep feedforward neural networks
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• All you need is a good init
• Deep Residual Learning for Image Recognition
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Attention Is All You Need
• Fixup Initialization: Residual Learning Without Normalization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Norm matters: efficient and accurate normalization schemes in deep networks
• Three Mechanisms of Weight Decay Regularization
• Adam: A Method for Stochastic Optimization
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• L2 Regularization versus Batch and Weight Normalization
• Bag of Tricks for Image Classification with Convolutional Neural Networks

Other Sources

• How to initialize deep neural networks? Xavier and Kaiming initialization
• Wikipedia LogSumExp
• Python decorators explained
• Understanding Xavier Initialization In Deep Neural Networks

Credits

• This interesting yet challenging project would not have been possible without the knowledge from fasi.ai videos.
• Lots of great OOP practice in my implementation were learned from the glorious PyTorch source code.
• If I were to make this documentation's front-end from ground up as well, you wouldn't be able to find this page. Thank you Xiaoying Riley for the PrettyDocs documentation template.