Compression Is Intelligence: What Cross-Entropy Really Measures (2026)

Cross-entropy measures prediction error in bits. Learn how Shannon entropy, data compression, and the training loss of a large language model turn out to be the same idea.