Gateway to ML Research

Table of contents:

• Machine Learning
• Deep Learning
• Reinforcement Learning
• NLP
• Computer Vision
• Generative Modeling
• Causal Inference
• Graph Neural Networks

• Linear Algebra
• Optimization
• Statistics & Probability

• Mathematics (General)
• Theoretical CS
• Algorithms
• Scientific Computing
• Information Theory
• Operation Research

• Jax
• Pytorch
• TensorFlow
• Julia
• Numpy
• Pandas
• Vim
• Linux
• Systems

• Mata-Resources
• Blog
• Newsletter

Machine Learning

• Mathematics for Machine Learning Book
• An Introduction to Statistical Learning Book
• Foundations of Applied Mathematics Book
• Foundations of Data Science Book
• Linear/non-Linear Regression model and Artificial Neural Network Github
• Nearest Neighbour style interpretations of Tree Ensembles Article
• PATTERNS, PREDICTIONS, AND ACTIONS A story about machine learning Book
• FAIRNESS AND MACHINE LEARNING Book
• Understanding Machine Learning: From Theory to Algorithms Book
• Learning Theory from First Principles Book
• A Course in Machine Learning Book
• Introduction to Machine Learning Book
• The HundredPage Machine Learning Book Book
• Machine Learning Glossary HTML Page
• Interpretable Machine Learning A Guide for Making Black Box Models Explainable HTML Page
• A Note on Machine Learning Methods Book
• Machine Learning Theory Course
• Introduction to Machine Learning Interviews Book HTML Page
• Probabilistic Machine Learning: An Introduction: 2nd Edition Book
• Machine Learning Mastery HTML Page
• CS229: Machine Learning Web Page
• A Gentle Introduction to Tensors Book
• MACHINE LEARNING A First Course for Engineers and Scientists Book
• AI Explorables: Big ideas in machine learning, simply explained Web Page
• CS 5785 Applied Machine Learning by Volodymyr Kuleshov at Cornell Tech Course
  • Video Series of the course Video
• Machine Learning From Scratch Github
• Machine Learning at Berkeley Reading List WebPage
• CS294-158-SP20 Deep Unsupervised Learning by Pieter Abbeel(UC Berkeley) Course
• Technion EE 046202 - Unsupervised Learning and Data Analysis Github + Course
• ICML 2020 Tutorial on Submodular Optimization: From Discrete to Continuous and Back WebPage
• 10-708 PGM - Probabilistic Graphical Models by Eric Xing(CMU) Course
• The Unsupervised Reinforcement Learning Benchmark (UC Berkeley) WebPage
• Mathematics for Machine Learning (UC Berkeley) Paper
• The Roadmap of Mathematics for Machine Learning Blog
• Machine Learning: The Basics by Alexander Jung Book
• Learning from Data by Caltech (2012) Course
• Algorithmic Aspects of Machine Learning by Ankur Moitra (MIT) Book
• Submodularity In Machine Learning and Artificial Intelligence Paper
• Harvard CS 229br: Advanced Topics in the theory of machine learning Course
• Harvard CS197: AI Research Experiences Course

Deep Learning

• Deep Learning by Yann LeCun HTML Page
  • NYU Deep Learning Video
• Hardware Architectures for Deep Neural Networks PowerPoint
  • Tutorial on Hardware Accelerators for Deep Neural Networks Web Page
• CS231n: Convolutional Neural Networks for Visual Recognition HTML Page
• Dive into Deep Learning HTML Page
• Getting started with Random Matrices: A Step-by-Step Guide Article
• What I Wish Someone Had Told Me About Tensor Computation Libraries Article
• DEEP GRAPH LIBRARY HTML Page
• Deep learning theory lecture notes HTML Page
• Neural Networks and Deep Learning HTML Page
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches Paper
• Deep Learning HTML Page
• Theory of Deep Learning Book
• The Modern Mathematics of Deep Learning Paper
• Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5 Article
• Adagrad Algorithm Explained and Implemented from Scratch in Python Video
• Attention Is All You Need - Annotated Paper + Paper Summary Paper + Article
• Deep Learning's Most Important Ideas - A Brief Historical Review Article
• Introduction to Deep Learning Course
• A Beginner's Guide to the Mathematics of Neural Networks Paper
• Physics-based Deep Learning HTML Page
• The Fourier transform is a neural network Web Page
• Deep Learning Papers Reading Roadmap Github
• labml.ai Deep Learning Paper Implementations Github
• The Matrix Calculus You Need For Deep Learning Web Page
• The Mathematical Engineering of Deep Learning Web Page
• The Mathematical Foundations of Manifold Learning by Luke Melas-Kyriazi(Harvard): Undergraduate Thesis Paper
• MiniTorch: diy teaching library for machine learning engineers by Sasha Rush(Cornell Tech) WebPage
• Yet another backpropagation tutorial by Boaz Barak WebPage
• EINSUM IS ALL YOU NEED - EINSTEIN SUMMATION IN DEEP LEARNING Blog
• Deep Learning Interviews: Hundreds of fully solved job interview questions from a wide range of key topics in AI Paper
• Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation Paper
• Applications of Deep Neural Networks with Keras Paper
• Deep Learning for Molecules and Materials Blog
• CS 7150: Deep Learning by Northeastern University Course
• 18.408 Theoretical Foundations for Deep Learning by Ankur Moitra (MIT) Course

Reinforcement Learning

• CS234: Reinforcement Learning Course
• CS 285 at UC Berkeley Deep Reinforcement Learning Course
• Simple Reinforcement Learning with Tensorflow series Article
• OpenAI Spinning Up HTML Page
• Introduction to Reinforcement Learning with David Silver Video + Slides
• REINFORCEMENT LEARNING AND OPTIMAL CONTROL Web Page
• Learning by Reinforcement and Optimal Control Course
• CS 598 Statistical Reinforcement Learning Course
• A Free course in Deep Reinforcement Learning from beginner to expert HTML Page
• Thomas Simonini Medium Articles Article
• Reinforcement Learning: Theory and Algorithms Book
• Offline Reinforcement Learning Workshop Web Page
• DEEP REINFORCEMENT LEARNING Paper
• IntroRL A course on reinforcement learning Course
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems Paper
• Deep Reinforcement Learning - Julien Vitay HTML Page
• RLpapers: List of Reinforcement Learning papers and code Web Page
• Lil'Log: Lilian OpenAI Researcher Blog Web Page
• Reinforcement Learning Course at ASU, Spring, 2021 Video
• Investigating Variational Gaussian Process State-Space Models with Gaussian Likelihood Paper
• Bandit Algorithms Book
• CS 498 Reinforcement Learning (S21) by Nan Jiang at UIUCCourse
• Comp-767 Reinforcement Learning by Doina Precup at McGill Course
• Introduction to Multi-Armed BanditsBook
• Reinforcement Learning as One Big Sequence Modeling Problem by Sergey Levine Paper
• Debugging RL, Without the Agonizing Pain Web Page
• CMU 15-889e Real Life Reinforcement Learning by Emma Brunskill Course
• Artificial Intelligence: Foundations of Computational Agents, 2nd Edition HTML Page
• Deep RL Bootcamp by UC Berkeley Course
• Deep Reinforcement Learning and Control by Ruslan Satakhutdinov Course
• Control Systems & Reinforcement Learning by Sean Meyn Book
• The Map of Control Theory Image
• An Intuitive Explanation of Policy Gradient Article
• The Policy of Truth by Ben Recht Web Page
• Let's Code Proximal Policy Optimization Video
• DQN in Pytorch from Scratch Video
• CS 277: Control and Reinforcement Learning by Roy Fox Course
• A Tutorial on Thompson Sampling Paper
• Reinforcement Learning (RL) Cheatsheet Github
• reinforcement-learning-discord-wiki Github
• RL links for tutorials/videos/blog/posts/open-source projects/libraries on twitter thread Twitter
• Minimal RL Algorithm Implementations Github
• RL-Adventure-2: Policy Gradients Github
• Algorithms for Decision Making Book
• Tutorial: Multi-Agent Learning by DeepMind Slides
• A Survey of Explainable Reinforcement Learning Paper
• Welcome to Transformer Reinforcement Learning (trl) Github

NLP

• An Introduction to Language Book
• The Mathematics of Language Book
• Wordplay Workshop: When Language Meets Games Web Page
• High Performance Natural Language Processing Slides
• Karthik Narasimhan's Princeton University Course Page Course
• NLP from Scratch with PyTorch, fastai, and HuggingFace Web Page
• Fine-tuning a BERT model Code
• Multi-label Text Classification using BERT – The Mighty Transformer Code
• Simple PyTorch Transformer Example with Greedy Decoding Code
• 10 NLP Resources: Books, papers, blog posts, lectures, hands-on courses. From Linguistics to Transformers (Twitter Thread) Tweet
• A Survey of Surveys (NLP & ML) Github
• NLP Tutorial Github
• A Hands-on Introduction to Natural Language Processing (NLP) Course + Video
  • Github link Github
• Resources for Understanding The Original Transformer Paper Reddit
• awesome-nlp:A curated list of resources dedicated to Natural Language Processing Github
• Awesome Deep Learning for Natural Language Processing (NLP) Github
• Natural Language Processing (guest lecture by Sasha Rush): Presentation Scribe Notes WebPage
• spaCy Tutorial to Learn and Master Natural Language Processing (NLP) Post
• NLP from Scratch with PyTorch, fastai, and HuggingFace Blog
• BERT Explained: A Complete Guide with Theory and Tutorial Post
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) Post
• The Annotated Transformer by Alexander M. Rush Paper
• A Primer in BERTology: What we know about how BERT works Paper
• NLP Tutorial: Implementation of NLP models Github
• 10 of the best resources to help you learn about the attention mechanism & Transformer network Twitter
• Attention is all you need" implementation from scratch in PyTorch. A Twitter thread Twitter
• CS224n: Natural Language Processing with Deep Learning (Stanford) Course
• Multi-head attention, GPT and BERT, Vision Transformer, and write these out in code by Misha Laskin Twitter
• CS224U: Natural Language Understanding | Spring 2021 Course
• CS11-711 Advanced NLP by CMU Course
• To Understand Language is to Understand Generalization by Eric Jang Blog
• Speech and Language Processing (3rd ed. draft) Book
• Dodrio Exploring transformer models in your browser! WebPage
• NLP Course | For You WebPage

Computer Vision

• CS 4476-B Computer Vision Course
• Beginner’s Guide to Computer Vision Article
• Pre-Trained Models for Computer Vision Github
• Computer Vision Notebooks Code
• Computer Vision Recipes Github

Generative Modeling

• Deep Generative Models CS236: Course Notes by Aditya Grover Web Page
• Generative Models Tutorial Github
• Probability Divergences and Generative Models Slides
  • Probability Divergences and Generative Models Video
• Tutorial - What is a variational autoencoder? WebPage
• CSC 2541: Differentiable Inference and Generative Models(Toronto) Course
• An Introduction to Autoencoders by Umberto Michelucci Paper
• Regression, Fire, and Dangerous Things (3/3): Thinking Like a Probability Distribution WebPage
• Generative or Discriminative? Getting the Best of Both Worlds Paper
• On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes Paper
• NIPS 2016 Tutorial: Generative Adversarial Networks Paper
• Reducing the Dimensionality of Data with Neural Networks Paper
• Tutorial on Variational Autoencoders Paper
• Normalizing Flows Tutorial, Part 1: Distributions and Determinants Blog
• Normalizing Flows by Adam Kosiorek Blog
• Indtroduction to Normalizing Flows by Krzysztof Kolasinski Slides
• Awesome Normalizing Flows: A list of awesome resources for understanding and applying normalizing flows (NF) Github

Causal Inference

• Awesome Causality: Resources Related to Causality HTML Page
• Causal Inference: What if Book
• Elements of Causal Inference Foundations and Learning Algorithms Book
• ML beyond Curve Fitting: An Intro to Causal Inference and do-Calculus Web Page
• Causal inference in statistics: An overview by Judea Pearl Paper
• Why There is not Statistical Test for Confounding, Why Many Think There is, and Why They are Almost Right by Judea Pearl Paper
• Causal Inference for The Brave and True HTML Page
• Counterfactual Data-Fusion for Online Reinforcement Learners by Judea Pearl Paper
• The Art and Science of Cause and Effect by Judea Pearl Paper
• Towards Causal Reinforcement Learning by Elias Bareinboim Slides
• List of Causal Inference Papers from COGNITIVE SYSTEMS LABORATORY LAB (Judea Pearl) WebPage
• Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases Paper

Graph Neural Networks

• Resources to help you learn and keep up-to-date with GNNs Tweet
• awesome-equivariant-network Github
• Theoretical Foundations of Graph Neural Networks Video
• GAT - Graph Attention Network (PyTorch) Github
• GDL Course Course
• Math Behind Graph Neural Networks WebPage
• Graph neural networks tutorial in pytorch Github
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges Paper
• A Comprehensive Survey on Graph Neural Networks Paper
• Graph Neural Networks: Models and Applications Slides
• Graph Representation Learning Book William L. Hamilton, McGill University Notes Course
• Stanford CS224W TA Notes WebPage
• Tutorial 7: Graph Neural Networks from UvA Deep Learning Course Web Page

Linear Algebra

• Introduction to Applied Linear Algebra Vectors, Matrices, and Least Squares Book
• Linear Algebra by Jim Hefferon Book
• Introduction to Linear Algebra for Applied Machine Learning with Python HTML Page
• Sketching as a Tool for Numerical Linear Algebra Book
• Linear Algebra Done Right Book
• Interactive Linear Algebra HTML Page
• The Art of Linear Algebra Github
• Linear Algebra Review and Reference Paper
• Linear Algebra and Matrix Calculus Review Slides
• Introduction to Linear Algebra by Gilbert Strang Book

Optimization

• An overview of gradient descent optimization algorithms by SEBASTIAN RUDER Article
• Convex Optimization by Stephen Boyd & Lieven Vandenberghe Book
• Convex Optimization: Algorithms and Complexity Book
• Convex Optimization by Ryan Tibshirani at CMU Course
• Algorithms for Optimization Book
• A Survey of Optimization Methods from a Machine Learning Perspective Paper
• CS 295 - Optimization for Machine Learning by Ioannis Penageas at UC Irvine Course
• EE 227C (Spring 2018) Convex Optimization and Approximation by Mortiz Hardt at UC Berkeley Course
• Convex Optimization Overview Paper
• Recent Advances in Non-Convex Optimization and its Implications to Learning by Anima Anadkumar Slides
• ECE 236A: Linear Programming(UCLA) Course
• ECE 236B: Convex Opimization(UCLA) Course
• ECE 236C: Optimization methods for large-scale systems(UCLA) Course
• ICML 2020 Tutorial on Submodular Optimization WebPage
• Combinatorial Optimization: Theory and Algorithms Book

Statistics and Probability

• The Elements of Statistical Learning Book
• Statistics 210A: Theoretical Statistics (Fall 2021) at UC Berkeley Course
  • Topics within the Syllabus Paper
• Prob 140: Probability for Data Science HTML Page
• The Language. Probabilistic Logic Programming HTML Page
• An Introduction to Probabilistic Programming Paper
• Statistics for Hackers Slides
• Foundations of Linear and Generalized Linear Models Book
• Stats 318 Modern Markov Chains at Stanford Course
• High-Dimensional Probability: An Introduction with Applications in Data Science by Roman Vershynin Book
• STAT 205A (= MATH 218A): Probability Theory at UC Berkeley Course
• Probability: Theory and Examples Book
• Seeing Theory: A Visual Introduction to Probability & Statistics HTML Page + Interactive
• Probability Cheatsheet Slides
• A Note on Probability Theory by Ying Nian Wu Book
• BIOSTAT M280/BIOMATH 280/STAT M230: Statistical Computing at UCLA Book
• The Probability and Statistics Cookbook Book
• An Introduction to Probabilistic modeling Slides
• A Concrete Introduction to Probability (using Python) Github
• Introduction to Probability for Data Science by Stanley H. Chan Book
• Review of Probability Theory Paper
• 36-401 Modern Regression(CMU) Course
• Regression, Fire, and Dangerous Things (1/3) WebPage
• Statistical Rethinking (2022 Edition) Github + Course
• Think Bayes 2 HTML

Mathematics

• SOME FUNDAMENTAL THEOREMS IN MATHEMATICS by OLIVER KNILL Book
• Upper-Division Course Notes: Numerical Analysis, Optimization, Group Theory, and Rings/Fields Course
• Applied Discrete Structures Book
• HOMOTOPY TYPE THEORY Book
• A computational perspective on set theory by Terence Tao Blog
• Set Theory and Foundations of Mathematics Web Page
• Set Theory in Computer Science A Gentle Introduction to Mathematical Modeling Book
• Set Theory Symbols Web Page
• The Web of Mathematics — Interactive Chart Article
• Mathematics for Computer Science: MIT Courseware Course + Book
• Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning Book
• Graph Theory and Additive Combinatorics by Yufei Zhao(MIT) Book
• 21-122 Integration and Approximation(CMU) Course
• An invitation to General Algebra and Universal Constructions by George M. Bergman Book
• WHAT IS GOOD MATHEMATICS? by Terence Tao Paper
• CALCULUS MADE EASY BY SILVANUS P. THOMPSON HTML

Theoretical CS

• Introduction to Theoretical Computer Science HTML Page
• Unravelling Complexity: The Life and Work of Gregory Chaitin: Chapter 1 (Preview) Book
• Computational Complexity: A Modern Approach Book
• 15-251 Great Ideas in Theoretical Computer Science(CMU) Course
• Why Philosophers Should Care About Computational Complexity Paper
• Theoretical Computer Science Cheat Sheet Paper
• Essential Coding Theory Book

Algorithms

• Design and Analysis of Algorithms: Blog by Rashid Bin Muhammad Web Page
• The Algorists HTML Page
• Cracking the Coding Interview (6th edition) Python Solutions Github
• Algorithms by Jeff Erickson at UIUC Book
• Problem Solving with Algorithms and Data Structures using Python HTML Page
• Big O Notation - explained as easily as possible Article
• Codeforces: Algorithm Category Web Page
• Red Blob Games: Interactive visual explanations of math and algorithms HTML Page + Interactive
• The Algorithms - Python Github
• A guide to learning algorithms Github
• Competitive Programmer’s Handbook Book
• Intro Data Structures(CMU) Course
• Dave Mount's Data Structures Lecture Notes Paper
• Dave Mount's Algorithm Lecture Notes:Design and Analysis of Computer Algorithms Paper
• 15-859: Algorithms for Big Data by David Woodruff(CMU) Course
• CS271 RANDOMNESS & COMPUTATION (UC Berkeley) Course
• CS 270. Combinatorial Algorithms and Data Structures (UC Berkeley) Course
• CS 224: Advanced Algorithms by Jelani Nelson Course
• Composing Programs HTML

Scientific Computing

• Neuromatch Academy Web Page
• Mathematical Tools for Neuroscience (Neurobio 212 at Harvard) Github
• Computational Linear Algebra for Coders by Rachel Thomas Github
• Introduction to Computational Thinking at MIT Course
• COMPUTATIONAL ALGEBRAIC TOPOLOGY LECTURE NOTES Book
• COS 302 / SML 305: Mathematics for Numerical Computing and Machine Learning at Princeton Course
• Quantitative Economics with Python HTML Page
• Computational Optimal Transport Book
• Bayesian Modeling and Computation in Python HTML

Information Theory

• Information Theory: A Tutorial Introduction Paper

Operation Research

• Operations Research APPLICATIONS AND ALGORITHMS Book

Jax

• ML Optimizers from scratch using JAX Github
• Curated list of awesome JAX libraries, projects, and other resources Github

Pytorch

• Deep Learning with Pytorch Video
• Pytorch Tutorial Github
• PyTorch internals Post
• PyTorchZeroToAll Github
• PYTORCH LIGHTNING Tutorial WebPage

TensorFlow

• TensorFlow Examples Github
• Learn TensorFlow and deep learning, without a Ph.D by GOOGLE Video +Slides
• Wrap up of Advent of Code 2021 in pure TensorFlow WebPage

Julia

• Parallel Computing and Scientific Machine Learning Video
• Statistics with Julia: Fundamentals for Data Science, Machine Learning and Artificial Intelligence Github

Numpy

• Scipy Lecture Notes using numpy HTML Page
• From Python to Numpy HTML Page
• Confetti.ai numpy HTML Page
• Numpy Internals Video
• A Visual Intro to NumPy and Data Representation Post
• 101 NumPy Exercises for Data Analysis (Python) Post
• Numpy Tools Github

Pandas

• pandas: powerful Python data analysis toolkit Book

Vim

• A Vim Guide for Advanced Users Web Page
• Learn Vim (the Smart Way) Github

Linux

• Comprehensive Linux Cheatsheet Web Page

Systems

• CS 149 PARALLEL COMPUTING (Stanford) Course
• CS 143 Compilers Course

Meta Resources

• Colin S. Gordon's Electronic Resources Web Page
• A Guide to Machine Intelligence Research Institute Web Page
• From-0-to-Research-Scientist-resources-guide Github
• Programming, Math, Science Github
• Blogs and websites on Machine Learning and Deep learning Github

Blog

• Christopher Olah's Blog: Various DL Tutorials Blog
• Jay Alammar: Visualizing machine learning one concept at a time Blog
• A List of Most Theory Blogs Blog
• Google AI Blog Blog
• OpenAI Blog Blog
• Twitter Blog Blog
• ML @Berkeley Blog
• DeepMind Blog Blog

Newsletter

• Machine Learning Ops Roundup Newsletter
• Deep Learning Weekly Newsletter
• The Gradient Newsletter
• IMPORT AI Newsletter
• Paper with Code Newsletter Newsletter
• NLP News By Sebastian Ruder Newsletter
• Davis Summarizes Papers Newsletter