training.md

training

nccl

• NCCL 2.0
• NVIDIA/nccl
• NCCL Installation Guide
• NVIDIA Collective Communication Library (NCCL) Documentation

data parallel

• 分布式训练的通讯原语
• 利用共享内存实现比NCCL更快的集合通信
• 分布式训练的一些问题
• GPU&CUDA多机通信
• 分布式机器学习之——Spark MLlib并行训练原理
• 一文读懂「Parameter Server」的分布式机器学习训练原理
• Visual intuition on ring-Allreduce for distributed Deep Learning
• Bringing HPC Techniques to Deep Learning
• Horovod知识储备：将HPC技术带入深度学习之中
• MPI，OpenMPI 与深度学习

open source

• dmlc/ps-lite
• baidu-research/baidu-allreduce
• baidu-research/tensorflow-allreduce
• horovod/horovod
• bytedance/byteps

todo

• Data Parallelism VS Model Parallelism in Distributed Deep Learning Training
• Model Parallelism in Deep Learning is NOT What You Think
• Data parallel and model parallel distributed training with Tensorflow