Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling
As growing compute budgets outpace the growth of high-quality training data, large language models must be trained with both compute and data constraints in mind. Unfortunately, existing prescriptions for data-constrained scaling suffer from two important limitations.
By Ethan Baron, Shmuel Berman, Anirudh Satheesh, Akshay Vegesna, Samip Dahal, Andrew Gordon WilsonarXiv cs.LG
Read the full story
Originally published by arXiv cs.LG on Oct 30, 2026.
Comments
Sign in to join the discussion.