Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success

Abstract

Many jailbreak attack research papers report attack success rates for a limited number of parameter settings, even though there are many combinations of parameter settings that could be used. Further, when new jailbreak papers are released, they often benchmark results against single configurations of existing attacks. This position paper argues such practices are fundamentally insufficient for characterising the threat posed by parameterised jailbreak attacks, and comparing attacks. Most jailbr

Categories

Framework mappings

Suggested from the entry's categories.

@misc{maple2026singleconfiguration,
  title = {{Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success}},
  author = {Carsten Maple and Abhishek Kumar and Riya Tapwal},
  year = {2026},
  month = may,
  eprint = {2605.09070},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.09070}
}