Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

Large neural networks are commonly trained across many workers, and communication between them is a major cost. Workers therefore take several local optimization steps before sending their net parameter displacements, or pseudo-gradients, to an outer optimizer, which often applies Nesterov…