Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages

This paper explores knowledge distillation (KD) of multilingual pre-trained encoder-decoder and large language models for machine translation (MT). We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam…