Muon Makes the First Mixture-of-Experts Layer Harder to Balance
Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router.
By Marcos HernanzarXiv cs.LG
Read the full story
Originally published by arXiv cs.LG on Oct 30, 2026.
Comments
Sign in to join the discussion.