Broken scale symmetries in undercomplete linear autoencoders

Neural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD).