The Price of Correlated Tests: How Strict Should a Model Release Gate Be?

Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is.