Phish or Legit: Do LLMs Know When NOT to Cry Wolf?

What happened

Anio Joseph conducted a Kaggle benchmarking challenge to evaluate how well LLMs can distinguish between phishing threats and legitimate messages. The benchmark includes 40 items, with 20 threats and 20 benign messages, some of which are designed to look suspicious. The models were tested on their ability to assign threat probabilities, identify the correct 'tell', and maintain calibration.

Why it matters

The results highlight the importance of not just detecting threats but also avoiding false positives, which can lead to alert fatigue. Calibration and explainability are crucial for trust and operational efficiency in security operations centers. The findings also suggest that smaller models can perform as well as larger ones in specific tasks, which has implications for cost and resource allocation.

• Accuracy alone is not sufficient; calibration and explainability are essential for security applications.
• False positives can be a significant operational burden in SOC environments.
• Smaller models like Gemini 3.7 Flash can perform as well as larger models in specific tasks.
• The 'Tell' accuracy is more challenging than the binary decision, indicating a need for better explainability.
• The next frontier in LLM evaluation is not just detection but also confidence and explanation.

The bigger picture

The benchmark results indicate that LLMs are becoming highly accurate in threat detection, but their calibration and ability to explain decisions are critical for real-world deployment. The findings suggest that smaller models can be effective for specific tasks, reducing costs and resource requirements.

This benchmarking challenge demonstrates that LLMs are becoming highly accurate in threat detection, but their calibration and ability to explain decisions are critical for real-world deployment. The results suggest that smaller models can be effective for specific tasks, reducing costs and resource requirements.

Technical details

• Balanced Accuracy metric combines threat recall and legitimacy specificity.
• Brier Score measures the calibration of confidence levels in predictions.
• Models tested include Gemini 2.5 Pro, GPT-5.4, Gemini 3.7 Flash, and Claude Sonnet 4.5.
• The benchmark includes 20 threats and 20 legitimate messages, some of which are designed to look suspicious.

FAQ