Know a great AI tool or AI news source? Submit it to AiByts
2 min read

Muon Makes the First Mixture-of-Experts Layer Harder to Balance

Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router.

By Marcos HernanzarXiv cs.LG

Read the full story

Originally published by arXiv cs.LG on Oct 30, 2026.

Read on arXiv cs.LG(opens in a new tab)

Comments

Sign in to join the discussion.

No comments yet — be the first to share your thoughts.