Rank Measures of Fine-Tuning Updates Need a Noise Baseline

Studies of what fine-tuning does to a language model often summarise the weight update $\Delta W$ by its rank. We compare the rank measures used for this against the same measures on a random matrix of the same shape, and against two harder references: the pretrained weights, and matched fine-tunes…