Multi-GPU MoE Inference with Adaptive Expert Offloading
Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand.
arXiv - cs.AI2 min read