Guardrails for autonomous agents, self-modification prevention, and containment
Newest first · 4 reviewed on this page
Search insteadTaein Lim, Seongyong Ju, Munhyeok Kim +2
Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, each agent exhibits an attack-selection bias,…
Chiyu Zhang, Huiqin Yang, Bendong Jiang +8
The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execute dangerous OS-level operations with irreversible…
Hardik Goel
Tool-enabled AI agents are increasingly deployed in cloud-hosted environments and offered as services, where they perform side-effecting operations through privileged tools within execution environments. While such agents enable powerful automation, the security implications of…
Yanting Wang, Chenlong Yin, Ying Chen +1
Long-context large language models (LLMs)-for example, Gemini-3.1-Pro and Qwen-3.5-are widely used to empower many real-world applications, such as retrieval-augmented generation, autonomous agents, and AI assistants. However, security remains a major concern for their…
Yukun Jiang, Yage Zhang, Michael Backes +2
Large language models (LLMs) have evolved into autonomous agents that rely on open skill ecosystems (e.g., ClawHub and Skills.Rest), hosting numerous publicly reusable skills. Existing security research on these ecosystems mainly focuses on vulnerabilities within skills, such as…
Yuntao Wang, Jianle Ba, Han Liu +5
The rapid evolution of large language model (LLM)-driven autonomous agents has given rise to OpenClaw, a new class of open-source agent frameworks that operate as continuously running, skill-augmented systems with persistent memory, multi-channel interaction, and high degrees of…
Doohee You
The expansion of Multimodal Large Language Models (MLLMs) and their integration into autonomous agentic workflows has introduced a non-stationary attack surface. Empirical observations indicate that adversaries employ progressive, cross-modal perturbations that evade…
Tyler Akidau, Tyler Rockwood, Johannes Brüderl +1
AI agents are increasingly expected to operate as digital employees: accessing enterprise data, making decisions, and taking actions autonomously. But agents are simultaneously less predictable than humans -- prone to hallucination, misinterpretation, and adversarial…
Manvendra Modgil
As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential. We study this timing problem using a continuous 18-dimensional affective-dynamics engine (HEART) as a…
Harshil Patel, Kunal Pai
As the Model Context Protocol (MCP) standardizes tool-calling for autonomous agents, it introduces a critical, unexamined attack surface: the error-handling loop. We hypothesize that tool error messages possess implicit authority, triggering corrective reasoning modes that…
Yuguang Zhou, Xunguang Wang, Pingchuan Ma +3
LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents. However, we reveal that the very reasoning and task-following capabilities enabling this protection introduce a novel vulnerability: attackers can…
Anupam Joshi, Tim Finin, Karuna Pande Joshi +1
Autonomous agentic AI systems driven by Large Language Models (LLMs) introduce a new class of security, privacy, and compliance challenges: an agent that can invoke tools, manipulate data, install software, and coordinate with peer agents across organizational boundaries must be…
Zhen Zhao, Yu Zhang, Yanpeng Zhu +4
Traditional OS security models based on "resource exposure plus permission checks" face structural challenges as LLM-driven autonomous agents acquire capabilities for planning, tool use, network access, and code execution. Once an agent runtime is compromised through prompt…
Seyed Bagher Hashemi Natanzi, Bo Tang
Large language models are no longer only text generators. They are increasingly embedded in retrieval pipelines, enterprise assistants, coding environments, robotic systems, security-operation workflows, and autonomous agents that can read private data, call tools, write files,…
Jimmy Laurence Rippin, Simon C. Marshall, David Demitri Africa +1
Increasingly autonomous agentic AI systems pose novel multi-agent risks, such as secret collusion via covert communication channels. The natural defence to these collusion attempts is to monitor plain-text communication, but the efficacy of monitors has been called into doubt by…
Shilin Ou, Yifan Xu, Luyao Zhang
As agentic AI systems are increasingly applied to cyber-physical environments, their evaluation requires assessment of both task performance and trustworthiness. In decentralized energy markets, autonomous agents may improve market utility, but may also exploit invalid physical…
Amin Beheshti, Rong N. Chang, Boualem Benatallah +7
The rapid emergence of LLM-powered autonomous and semi-autonomous agents is reshaping software systems from static, request-response components into goal-directed, adaptive, and tool-using computational actors. As these agents move from isolated cognitive prototypes into complex…
Andreas Happe, Jürgen Cito, Jasmin Wachter
LLM-driven autonomous agents are reshaping offensive security. Unlike traditional penetration-testing tooling -- deterministic, narrowly scoped, and operated by trained practitioners -- agentic security tools exhibit \textit{indeterminacy} along three independent dimensions.…
Shasha Yu, Fiona Carroll, Barry L. Bentley
Large language models (LLMs) serving as planners in tool-using autonomous agents introduce dynamic reliability risks in multi-turn execution. While single-turn safety mechanisms are relatively mature, extended interactions reveal structural vulnerabilities where initial…
Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis +11
As autonomous agents powered by foundation models are increasingly integrated into social and economic systems, understanding the principles governing their collective behavior is essential for ensuring safety and cooperation. Classical game theory, the dominant framework for…
Md Jafrin Hossain, Mohammad Arif Hossain, Nirwan Ansari
Large Language Models (LLMs) have undergone a shift from stateless conversational interfaces to autonomous agents capable of multi-step planning, tool invocation, code execution, and maintaining persistent memory. When these agents operate with real-world privileges---calling…
Albus W. Ng, Yi Han, Jusheng Zhang +1
The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety…
Sanket Badhe, Priyanka Tiwari, Jonghyun Chung
Large Language Models (LLMs) increasingly act as autonomous agents executing complex, long-running procedural skills. Existing agent runtimes maintain execution by continually appending observations, actions, and intermediate reasoning traces to an ever-growing conversation…
Zhongjian Hu, Peng Yang, Dongmei Yang +1
Large Language Models (LLMs) have demonstrated strong capabilities in knowledge-based Visual Question Answering (VQA). However, existing prompt construction methods are often rigid and fail to fully exploit the reasoning potential of LLMs. To address this limitation, we propose…
Behnam, Mohammadkhani, Atul Khekade +1
Autonomous agents can already pay per request: HTTP-native protocols such as x402 let an agent sign a stablecoin authorization and receive a resource in the same round trip. That model is atomic and final, which suits metered access and fails commerce: a purchase made on a…
Sebastiano Nordio, Michele Lotto
The proliferation of highly capable open-weight Small Language Models (SLMs) democratizes access to advanced cybersecurity capabilities, posing a escalating risk as these models can bypass proprietary API guardrails when deployed locally. However, SLMs deployed as autonomous…
Dingjie Song, Hanrong Zhang, Dawei Liu +10
Command-line coding agents (e.g., Claude Code, Gemini CLI) can already read and write files and sustain long sessions, yet end-to-end research still fragments across chat tools, IDEs, terminals, and writing environments, and the decisions that make it auditable are rarely…
Maurice E. Dawson, A. B. Ben Ayed, Samson Quaye
Industry 5.0 places autonomous agents inside its human–agent collaborative loop, resting its human-centricity pillar on an assumption of trustworthy collaboration that has not been examined critically. Agentic artificial intelligence has moved from research demonstration to…
Tobin South, Subramanya Nagabhushanaradhya, A. Dissanayaka +18
The rapid rise of AI agents presents urgent challenges in authentication, authorization, and identity management. Current agent-centric protocols (like MCP) highlight the demand for clarified best practices in authentication and authorization. Looking ahead, ambitions for highly…
Edoardo Allegrini, Ananth Shreekumar, Z. B. Celik
Agentic AI systems, which leverage multiple autonomous agents and large language models (LLMs), are increasingly used to address complex, multi-step tasks. The safety, security, and functionality of these systems are critical, especially in high-stakes applications. However, the…
P. Zambare, Venkata Nikhil Thanikella, Ying Liu
When combining Large Language Models (LLMs) with autonomous agents, used in network monitoring and decision-making systems, this will create serious security issues. In this research, the MAESTRO framework consisting of the seven layers threat modeling architecture in the system…
Richard Fang, Rohan Bindu, Akul Gupta +2
Demonstrates that LLM agents can autonomously perform web hacking tasks including SQL injection, XSS, and CSRF attacks without human guidance.
John Yang, Carlos E. Jimenez, Alexander Wettig +4
Demonstrates autonomous coding agents that interact with computer interfaces to solve software engineering tasks, raising questions about agent containment.
Richard Fang, Rohan Bindu, Akul Gupta +1
Shows that LLM agents (GPT-4) can autonomously exploit real-world one-day vulnerabilities given CVE descriptions, achieving 87% success rate.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang +5
Demonstrates a continuously learning LLM agent in Minecraft that writes and executes code, highlighting autonomous operation and containment challenges.