The Null Is the Hard Part: Exact Tests for Memorization in Generative Models

Memorization audits of generative models read similarity scores against thresholds, with no null distribution, and the conclusions they support can be wrong.