Know a great AI tool or AI news source? Submit it to AiByts

Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

By /u/ifapostoReddit - r/ML

Read the full story

Originally published by Reddit - r/ML on Oct 27, 2026.

Read on Reddit - r/ML(opens in a new tab)

Comments

Sign in to join the discussion.

No comments yet — be the first to share your thoughts.