Zero Bubble Pipeline Parallelism
Systems and Performance
Deep Learning
Machine Learning
Read transcript · Original paper
AI-generated voices discussing a paper selected by Arjun Srivastava.
About this paper
Core idea is think about backward pass into two flows, one to compute grad wrt to parameters, and one to compute grad wrt to output of last layer, schedule so that you are always working instead of waiting (bubble).