Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

Abstract

Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by

Categories

Framework mappings

Suggested from the entry's categories.

@misc{liu2026beyond,
  title = {{Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces}},
  author = {Junlong Liu and Haobo Wang and Weiqi Luo and Xiaojun Jia},
  year = {2026},
  month = jul,
  eprint = {2607.00481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.00481}
}