Quantization vs Distillation: How Big AI Models Get Small
Frontier AI models are almost too big to use — a 70B model needs ~140 GB of memory just to hold its weights. So how do these giants end up running on a single GPU, or even on a laptop?
Scrollypedia