CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak
Automated red teaming trains an attacker large language model (LLM) to elicit harmful responses from a victim LLM, but reinforcement learning can collapse attackers onto a narrow set of prompts when attack success is the only reward.
arXiv - cs.CL (NLP)2 min read