Know a great AI tool or AI news source? Submit it to AiByts

EMA

3 stories

Stories

  • Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]

    Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

    Reddit - r/ML1 min read
  • Teachers other than EMA for distillation ? [R]

    Self-distillation methods, see Self-Distillation Fine Tuning SDFT, usually use an EMA (Exponential Moving Average) teacher to get synced and stable targets.

    Reddit - r/ML1 min read

Showing 3 of 3 stories

You're all caught up.

AiByts Weekly Brief

Get the top AI news and tools straight to your inbox every week.

Weekly · AI news

Join 15,000+ AI enthusiasts