README.md

Session 9: Reducing Activation Recomputation in Large Transformer Models

EleutherAI ML Scalability & Performance Reading Group Session 9, which builds on the TP strategy introduced in Megatron-LM with some additional techniques: sequence parallelism and selective activation recomputation.

Presenter: Daniel Vega-Myhre

Papers:

1. Reducing Activation Recomputation in Large Transformer Models

Links:

• Recording
• Annotated paper