Repository Content:

• This repository contains good coding practices and ideas for practice.
• It provides examples of professional-level codebases for training neural networks.

Practice Approach:

• Practice reading code as if they are research papers to enhance your skills.
• Reading code in this manner is essential for improving your proficiency.

Example Codebases:

• Wide Resnet:
  • Example code for training wide residual networks.
  • Repository: https://github.com/szagoruyko/wide-residual-networks
• Robust Overfitting:
  • Example code for training in professional settings with a focus on robust overfitting.
  • Repository: https://github.com/locuslab/robust_overfitting
• Low Curvature Activations:
  • Example code for training with low curvature activations.
  • Repository: https://github.com/vasusingla/low_curvature_activations?tab=readme-ov-file
• Transformers:
  • Example code for training Transformers.
  • Repository: https://nlp.seas.harvard.edu/2018/04/03/attention.html

Learning Outcomes:

• After reviewing these codebases, you will gain insights into the fundamentals of professional neural network training.
• Topics covered include logging, checkpointing, early stopping, and learning rate scheduling.