Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code

Abstract

Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLM

Categories

Framework mappings

Suggested from the entry's categories.

@misc{zhang2026grammarconstrained,
  title = {{Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code}},
  author = {Yitong Zhang and Shiteng Lu and Jia Li},
  year = {2026},
  month = jun,
  eprint = {2606.11817},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.11817}
}