Folding Language Models into Themselves
Knowledge distillation usually trains a compact student from scratch, which needs a large amount of data even though the teacher's weights already encode most of what the student has to learn. In this work, we explore building the student directly from the teacher's weights.
By Ziyang Zhang, Yubin Jing, Yuyao Li, Haofan Wang, Yichen GongarXiv - cs.CL (NLP)
Read the full story
Originally published by arXiv - cs.CL (NLP) on Nov 16, 2026.
Comments
Sign in to join the discussion.