2 stories
Cross-entropy measures prediction error in bits. Learn how Shannon entropy, data compression, and the training loss of a large language model turn out to be the same idea.
Few days ago we had a talk by Gergely Neu, who presented his recent work:Gergely Neu Information-Theoretic Generalization Bounds for Stochastic Gradient DescentI'm writing this post mostly to annoy him, by presenting this work using super hand-wavy intuitions and cartoon figures. If this isn&
Showing 2 of 2 stories
You're all caught up.
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts
