Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization

Abstract

Jailbreak attacks on audio language models (ALMs) optimize audio perturbations to elicit unsafe generations, and they typically update the entire waveform densely throughout optimization. In this work, we investigate the necessity of such dense optimization by analyzing the structure of token-aligned gradients in ALMs. We find that gradient energy is highly non-uniform across audio tokens, indicating that only a small subset of token-aligned audio regions dominates the optimization signal. Motiv

Categories

Framework mappings

Suggested from the entry's categories.

@misc{fang2026sparse,
  title = {{Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization}},
  author = {Zheng Fang and Xiaosen Wang and Shenyi Zhang and Shaokang Wang and Zhijin Ge},
  year = {2026},
  month = may,
  eprint = {2605.04700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.04700}
}