LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

Abstract

The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execute dangerous OS-level operations with irreversible consequences. Existing benchmarks either evaluate safety at the semantic layer alone, missing physical-layer harms, or fail to isolate test cases, letting earlier runs contaminate later ones. We present LITMUS (LLM-agents In-OS

Categories

Framework mappings

Suggested from the entry's categories.

@misc{zhang2026litmus,
  title = {{LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments}},
  author = {Chiyu Zhang and Huiqin Yang and Bendong Jiang and Xiaolei Zhang and Yiran Zhao and Ruyi Chen and Lu Zhou and Xiaogang Xu and Jiafei Wu and Liming Fang and Zhe Liu},
  year = {2026},
  month = may,
  eprint = {2605.10779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.10779}
}