AlterNet: Adaptive-Depth Mixture-of-Experts with Learned Halting for Elastic Inference
Most LLM architectures have fixed compute demands determined at training time. We propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth system to make the extent of a forward pass dynamically adjustable.
arXiv - cs.AI2 min read