Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

Abstract

LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. Most safety benchmarks evaluate defenders against fixed attack pools collected before evaluation, single-turn or multi-turn. We present a 21-scenario benchmark for \emph{adaptive multi-round attacks against memoryless LLM defenders}: an autonomous LLM attacker observes prior defender responses and pivots across rounds, while each defender response is evaluated as a fresh interaction. Holding

Categories

Framework mappings

Suggested from the entry's categories.

@misc{jain2026adaptive,
  title = {{Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security}},
  author = {Devina Jain and David Hartmann and Chuan Li},
  year = {2026},
  month = jul,
  eprint = {2607.18063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.18063}
}