gap2 min read
GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection
Sparse Mixture-of-Experts (SMoE) models improve scalability by routing input tokens to only a subset of expert networks, but they often suffer from representation collapse, in which multiple experts converge toward similar functions and lose their intended diversity.
By Donghyun Jeong, Yoon-Yeong KimarXiv - stat.ML
Read the full story
Originally published by arXiv - stat.ML on Oct 22, 2026.
Comments
Sign in to join the discussion.