How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Looped transformers reuse one block of layers many times and are often trained by backpropagating through only the last k loops. We measure how much of the weight gradient this throws away, exactly and for every k from one backward pass, on nine looped language models, and report three findings.
By Aur Shalev MerinarXiv cs.LG
Read the full story
Originally published by arXiv cs.LG on Oct 22, 2026.
Comments
Sign in to join the discussion.