paperAugust 2026Unreviewed

Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Jiajia Li, Chaochao Lu, Qiaosheng Zhang

Abstract

Reliable jailbreak evaluation is essential for assessing LLM safety, but most existing studies rely solely on attack success rate (ASR) without accounting for its dependence on attack budgets, resulting in unfair comparisons across methods. Existing compute-aware evaluations reduce heterogeneous resources into FLOPs, which is difficult to estimate for black-box models and fails to capture resource-specific constraints. To provide a comparable evaluation basis, we introduce Fair-ASR, an evaluatio

Framework mappings

OWASP Top 10 for LLM Applications

LLM01Prompt Injection

MITRE ATLAS

AML.T0054LLM Jailbreak

Suggested from the entry's categories.

Cite

@misc{he2026fair,
  title = {{Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets}},
  author = {Zhida He and Xiaoyu Wen and Han Qi and Ziyuan Zhou and Peng Yu and Jiajia Li and Chaochao Lu and Qiaosheng Zhang},
  year = {2026},
  month = aug,
  eprint = {2608.17360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.17360}
}