Direct, indirect, and multi-turn prompt injection attacks
Newest first · 13 reviewed on this page
Search insteadWilliam Hackett, Lewis Birch, Stefan Trawicki +2
Large Language Models (LLMs) guardrail systems are designed to protect against prompt injection and jailbreak attacks. However, they remain vulnerable to evasion techniques. We demonstrate two approaches for bypassing LLM prompt injection and jailbreak detection systems via…
Zhilong Wang, N. Nagaraja, Lan Zhang +3
LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model’s behavior. Traditional defenses like input sanitization, guard models, and…
Jinyu Chen, Jixiao Yang, Ziyang Zeng +3
Large Language Model (LLM)-based multi-agent systems have demonstrated remarkable capabilities across diverse applications, yet they face critical security challenges including backdoor attacks, prompt injection, and privacy leakage. Existing defense mechanisms typically address…
S. Hossain, Ruksat Khan Shayoni, Mohd Ruhul Ameen +3
Prompt injection attacks represent a major vulnerability in Large Language Model (LLM) deployments, where malicious instructions embedded in user inputs can override system prompts and induce unintended behaviors. This paper presents a novel multi-agent defense framework that…
Pavan Reddy, Aditya Sanjay Gujral
Large language model (LLM) assistants are increasingly integrated into enterprise workflows, raising new security concerns as they bridge internal and external data sources. This paper presents an in-depth case study of EchoLeak (CVE-2025-32711), a zero-click prompt injection…
Hammad Atta, M. Baig, Yasir Mehmood +5
We introduce Cognitive Degradation as a novel vulnerability class in agentic AI systems. Unlike traditional adversarial external threats such as prompt injection, these failures originate internally, arising from memory starvation, planner recursion, context flooding, and output…
Xin Wei Chia, Jonathan Pan
Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, yet they remain vulnerable to adversarial manipulations such as jailbreaking via prompt injection attacks. These attacks bypass safety mechanisms to generate restricted or harmful…
Zimo Ji, Xunguang Wang, Zongjie Li +6
Large Language Model (LLM)-based agents with function-calling capabilities are increasingly deployed, but remain vulnerable to Indirect Prompt Injection (IPI) attacks that hijack their tool calls. In response, numerous IPI-centric defense frameworks have emerged. However, these…
Eranga Bandara, Amin Hass, Ross Gore +8
AI agent-based systems are becoming increasingly integral to modern software architectures, enabling autonomous decision-making, dynamic task execution, and multimodal interactions through large language models (LLMs). However, these systems introduce novel and evolving security…
Jiaming Hu, Haoyu Wang, Debarghya Mukherjee +1
Jailbreak attacks pose a serious challenge to the safe deployment of large language models (LLMs). We introduce CCFC (Core&Core-Full-Core), a dual-track, prompt-level defense framework designed to mitigate LLMs'vulnerabilities from prompt injection and structure-aware jailbreak…
Avihay Cohen
Large Language Model (LLM) based agents integrated into web browsers (often called agentic AI browsers) offer powerful automation of web tasks. However, they are vulnerable to indirect prompt injection attacks, where malicious instructions hidden in a webpage deceive the agent…
Michal Podpora, Marek Baranowski, Maciej Chopcian +2
Large Language Models with Retrieval-Augmented Generation are considered to be modern, chat-native interfaces to enterprise knowledge. However, deploying such systems safely requires precautions more advanced than input filtering. Numerous LLM-related security threats (including…
Dominik Schwarz
The security of Large Language Model (LLM) applications is fundamentally challenged by "formfirst" attacks like prompt injection and jailbreaking, where malicious instructions are embedded within user inputs. Conventional defenses, which rely on post hoc output filtering, are…
Yi Liu, Gelei Deng, Yuekang Li +6
First comprehensive survey of prompt injection attacks against LLM-integrated applications, categorizing attacks and defenses with a unified framework.
Eric Wallace, Kai Xiao, Reimar Leike +3
Proposes an instruction hierarchy for training LLMs to prioritize system prompts over user prompts over third-party content, as a defense against prompt injection.
Sam Toyer, Olivia Watkins, Ethan Adrian Mendes +9
Uses data from an online game (Tensor Trust) where players compete to craft prompt injections and defenses, creating a large dataset of human-generated attacks.
X. Suo
The critical challenge of prompt injection attacks in Large Language Models (LLMs) integrated applications, a growing concern in the Artificial Intelligence (AI) field. Such attacks, which manipulate LLMs through natural language inputs, pose a significant threat to the security…
Edoardo Debenedetti, Jie Zhang, Mislav Balunovic +3
Introduces AgentDojo, a framework for evaluating the security of LLM agents against prompt injection and other attacks in realistic tool-use scenarios.
Rodrigo Pedro, Daniel Castro, Paolo Molina +1
Demonstrates how prompt injection can be chained with traditional web attacks (SQL injection, XSS) in LLM-integrated applications.
Qiusi Zhan, Zhixiang Liang, Zifan Ying +1
Presents InjecAgent, a benchmark for evaluating indirect prompt injection attacks against LLM agents that use tools, showing most agents are highly vulnerable.
Jingwei Yi, Yueqi Xie, Bin Zhu +5
Provides a benchmark for indirect prompt injection attacks and evaluates several defense strategies including perplexity-based detection and sandwich defense.
Yupei Liu, Yuqi Jia, Runpeng Geng +2
Proposes defense mechanisms against prompt injection in LLM systems including isolation-based approaches, input/output filtering, and detection methods.
Sagiv Antebi, Noam Azulay, Edan Habler +3
Analyzes security risks of custom GPTs in the OpenAI GPT Store including prompt leakage, data exfiltration, and malicious GPTs.
Oleksandr Muliarevych
This article examines cybersecurity vulnerabilities in systems utilizing Language Model Interfaces, focusing on the challenges of building secure systems. It provides an overview of current interfaces and their associated risks. A key contribution is the design of a prompt…
DeadBits
Open-source scanner for detecting prompt injections using vector similarity, YARA rules, text classifiers, and canary tokens.
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra +3
Introduces indirect prompt injection attacks against LLM-integrated applications, demonstrating how adversaries can remotely control LLMs by injecting prompts into data sources the LLM retrieves.
Sander Schulhoff, Jeremy Pinto, Anaum Khan +7
Presents results from a global prompt hacking competition with 600K+ adversarial prompts, revealing systemic LLM vulnerabilities across multiple models and defense strategies.
Protect AI
Open-source tool designed to detect and prevent prompt injection attacks using multiple detection methods including heuristics, LLM-based analysis, and canary tokens.