LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

Chiyu Zhang, Huiqin Yang, Bendong Jiang, Xiaolei Zhang, Yiran Zhao, Ruyi Chen, Lu Zhou, Xiaogang Xu, Jiafei Wu, Liming Fang, Zhe Liu

Abstract

The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execute dangerous OS-level operations with irreversible consequences. Existing benchmarks either evaluate safety at the semantic layer alone, missing physical-layer harms, or fail to isolate test cases, letting earlier runs contaminate later ones. We present LITMUS (LLM-agents In-OS

Categories

Cite This Resource

@article{llmsec202600102,
  title = {LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments},
  author = {Chiyu Zhang and Huiqin Yang and Bendong Jiang and Xiaolei Zhang and Yiran Zhao and Ruyi Chen and Lu Zhou and Xiaogang Xu and Jiafei Wu and Liming Fang and Zhe Liu},
  year = {2026},
  url = {https://arxiv.org/abs/2605.10779},
}