CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Taein Lim, Seongyong Ju, Munhyeok Kim, Hyunjun Kim, Hoki Kim

Abstract

Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, each agent exhibits an attack-selection bias, disproportionately concentrating its efforts on a narrow subset of attack families regardless of prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evalua

Categories

Cite This Resource

@article{llmsec202600078,
  title = {CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios},
  author = {Taein Lim and Seongyong Ju and Munhyeok Kim and Hyunjun Kim and Hoki Kim},
  year = {2026},
  url = {https://arxiv.org/abs/2605.07830},
}