AI-specific threat models, attack taxonomies, and kill chains
Newest first · 10 reviewed on this page
Search insteadQi Wang, Chengcheng Wan, Jiangtao Wang
Large language models (LLMs) are increasingly deployed in safety-critical applications, yet jailbreak attacks can conceal harmful intent through role-playing, fictional scenarios, or seemingly benign motivations. Existing inference-time defenses may miss disguised attacks or…
Yixuan Liu, Zilong Zhen, Yin Wu +1
As Large Language Model (LLM) agents increasingly automate offensive operations across the cyber kill chain, their efficacy in complex local post-exploitation tasks remains inadequately quantified. Among these, Linux privilege escalation is a key step between initial access and…
Anna Gazani, Spyridon Kounoupidis, Panagiotis Katsaros +3
The integration of Large Language Models (LLMs) into Security Operations Centers (SOCs) streamlines threat intelligence but introduces critical vulnerabilities, notably indirect prompt injection via log poisoning. Adversaries exploit this vector to execute multistep…
Manit Kaushik, Ishir Bhardwaj, Pranav Gupta +2
JavaScript powers approximately 98.8% of all websites, making vulnerabilities in its code a significant security risk, yet existing detection approaches such as Static Application Security Testing (SAST) tools often fail to identify many real-world vulnerabilities when applied…
Linhai Ma, Rita El Hachem, Mahatab El Hajj +2
Crisis helplines assess suicide risk through structured interviews, a process that is slow and dependent on operator training and workload. Natural language processing could support risk assessment and call prioritization, but almost no work addresses Arabic-language helpline…
Vishwanath
Foundation models increasingly operate across modalities (vision, language, audio, and vision–language) and are deployed in decision-critical pipelines with tool use and retrieval. This expands the adversarial surface: small perturbations to images or audio can flip predictions,…
Meng-Lin Liu, Yao Yu, Tong Wu +2
As large language models become the front door to political information, what they refuse to discuss becomes a new instrument of information control. We argue that a model's guardrail encodes not a universal notion of harm but the political threat model of the state that governs…
Antonio Lara-Gutierrez, Juan Zamorano, J. A. Onieva
Honeypots are widely used as cyber-deception tools to study adversarial behaviour, yet their effectiveness is limited by a trade-off between realism and security risk. Low-interaction honeypots are easily detected, while high-interaction honeypots provide realistic data at the…
Albert Adusei Brobbey, Narayan P. Bhosale
Agentic artificial intelligence systems increasingly depend on persistent runtime memory, including vector databases, episodic memory stores, long‐term retrieval indices, and cloud‐to‐edge replicas. Existing security frameworks address prompt injection, data poisoning, and…
Wen-Jing Chen, Jie Cui, Wenjie Huang +3
While Large Language Models (LLMs) have achieved revolutionary advancements in natural language processing, their inherent vulnerability to prompt injection attacks has raised significant security concerns. Existing security detection approaches for LLM deployment often fail to…
Kang Chen, Xiu-Ze Zhou, Yuanhui Yu +4
Large Language Models (LLMs), now a foundation in advancing natural language processing, power applications such as text generation, machine translation, and conversational systems. Despite their transformative potential, these models inherently rely on massive amounts of…
Jiawei Liu, Jiacheng Guo, Tian Zhang +4
Foundation models are increasingly used for perception, reasoning, planning, and action generation in embodied agents, creating security risks that can propagate from digital inputs to physical behavior. Existing surveys often organize threats by mechanisms such as jailbreaks,…
Bhavesh B. Prajapati, Bhavya Shah
Large language model (LLM) applications now combine hidden prompts, retrieval pipelines, memory stores, content filters, tool calls, delegated identities, and downstream automation. Security reviewers are increasingly asked to assess such systems without access to source code,…
Wonbae Kim, Hee-Kyong Yoo, Nammee Moon
The deployment of Large Language Model (LLM)-generated SQL in Artificial Intelligence of Things (AIoT) systems introduces critical security risks, as prompt injection attacks can manipulate LLMs into producing unauthorized queries that expose sensitive data or execute…
Jian Zhao, Shenao Wang, Qingyang Wu +3
The widespread adoption of open source software (OSS) has introduced significant security risks, with malicious code poisoning attacks increasingly targeting public package registries and open-source platforms. Existing detection approaches, including heuristic-, learning-, and…
C. Thapa, Mohan Baruwal Chhetri, M. Grobler +2
Frontier artificial intelligence (FAI), encompassing large-scale, general-purpose AI systems, including large language models, multimodal foundation models, and agentic systems, is increasingly integrated into critical infrastructure (CI). This challenges long-standing security…
Sushma Mahadevaswamy
The rapid enterprise adoption of agentic artificial intelligence (AI) has introduced a category of security risk that existing cyber security frameworks were not designed to address. With 78 per cent of Fortune 500 companies projected to deploy agentic AI by 2026 and the global…
Baiqi Wu, Qing-Ming Li, Chun-Yi Zhou +2
Rapid AI development across industries raises pressing security and privacy risks. This work presents a unified comparison of large language models, AI agents, and embodied agents, introducing a taxonomy of risks spanning data, models, systems, content, and applications,…
Galal Eltayeb, Abdalilah Alhalangy
Abstract But now, given the AI revolution and increased interest in bringing virtual agents and assistants to life banks too are testing LLM-powered AI agents that may assist customers, explain and customize products as well as simplify operational work done by bank employees in…
Saurabh Sharma
Abstract Prompt injection—the manipulation of an LLM-based agent through adversarial content embedded in user input—poses a critical security risk in production agentic systems with access to sensitive datastores and code execution environments. Detection-based defenses (keyword…
Divya Pandey, Shivani Manchanda, Gangesh Pathak +1
Clinical large language model (LLM) agents are entering production hospital deployments, where they read longitudinal electronic health records (EHRs), retrieve evidence from clinical knowledge bases, and assist with summarization, dosing, triage, and guideline-based decisions.…
Zanhong Zheng, Jieming Liang, Mengqin Hu +3
Prompt injection detection is commonly studied as a static offline classification problem, yet deployed LLM systems face evolving attacks and distribution shift after deployment. Static detectors are therefore poorly matched to the threat model, while routing every input to a…
Pranav Bhatnagar
Large Language Models (LLMs) are increasingly deployed as persistent, interactive systems that retain information across user interactions. These memory mechanisms are designed to enhance personalization, task continuity, and operational efficiency. However, persistence…
James H. Senanu
The rapid integration of artificial intelligence (AI) systems into security-critical domains has introduced new vulnerabilities, exposing these systems to a growing spectrum of adversarial threats. Adversarial machine learning (AML) has emerged as a key area of research aimed at…
Harsh Verma
As artificial intelligence becomes woven into critical applications such as healthcare, finance, autonomous systems, and cybersecurity, adversarial threats to machine learning models have grown into one of the most pressing concerns in the field. Adversarial machine learning…
L. Ahmad, S. Agarwal, Michael Lampe +1
Red teaming has emerged as a critical practice in assessing the possible risks of AI models and systems. It aids in the discovery of novel risks, stress testing possible gaps in existing mitigations, enriching existing quantitative safety metrics, facilitating the creation of…
Ibrahim Adabara, Bashir Olaniyi Sadiq, Aliyu Nuhu Shuaibu +2
Agentic Artificial Intelligence systems, characterized by autonomous reasoning, memory augmentation, and adaptive planning, are rapidly reshaping technological landscapes. Unlike traditional AI or large language models, agentic AI integrates decision-making with persistent…
L'eo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru +9
We present DoomArena, a security evaluation framework for AI agents. DoomArena is designed on three principles: 1) It is a plug-in framework and integrates easily into realistic agentic frameworks like BrowserGym (for web agents) and $\tau$-bench (for tool calling agents); 2) It…
Yanxu Mao, Tiehan Cui, Peipei Liu +2
Large language models (LLMs) are rapidly evolving from single-modal systems to multimodal LLMs and intelligent agents, significantly expanding their capabilities while introducing increasingly severe security risks. This paper presents a systematic survey of the growing…
Ali Eslami, Jiangbo Yu
Agentic AI is increasingly being explored and introduced in both manually driven and autonomous vehicles, leading to the notion of Agentic Vehicles (AgVs), with capabilities such as memory-based personalization, goal interpretation, strategic reasoning, and tool-mediated…
P. Zambare, Venkata Nikhil Thanikella, Ying Liu
When combining Large Language Models (LLMs) with autonomous agents, used in network monitoring and decision-making systems, this will create serious security issues. In this research, the MAESTRO framework consisting of the seven layers threat modeling architecture in the system…
Hanbin Hong, Shuangqiao Wu, Shuya Feng +6
Large Language Models (LLMs) are increasingly used as interfaces to information, code, and real-world services, making prompt-level security failures a practical concern. Although jailbreak attacks, defenses, datasets, and automated judgers have advanced rapidly, evaluation…
Eranga Bandara, Amin Hass, Ross Gore +8
AI agent-based systems are becoming increasingly integral to modern software architectures, enabling autonomous decision-making, dynamic task execution, and multimodal interactions through large language models (LLMs). However, these systems introduce novel and evolving security…
Steve Wilson, OWASP LLM AI Security Team
The definitive OWASP guide identifying the top 10 most critical security risks in LLM applications, with descriptions, examples, and mitigation strategies.
OWASP GenAI Security Project
Identifies the top 10 security risks specific to agentic AI applications including excessive agency, unsafe tool execution, and inadequate oversight.
Aysan Esmradi, Daniel Wankit Yip, Chun Fai Chan
Surveys attack techniques across the LLM lifecycle including training, fine-tuning, and inference, with comprehensive mitigation strategies.
Bhagyajit Pingua, Deepak Murmu, Meenakshi Kandpal +4
Large language models (LLMs) have become transformative tools in areas like text generation, natural language processing, and conversational AI. However, their widespread use introduces security risks, such as jailbreak attacks, which exploit LLM’s vulnerabilities to manipulate…
Jing Cui, Yishi Xu, Zhewei Huang +3
Large Language Models (LLMs) have revolutionized artificial intelligence and machine learning through their advanced text processing and generating capabilities. However, their widespread deployment has raised significant safety and reliability concerns. Established…
MITRE Corporation
Knowledge base of adversarial tactics, techniques, and case studies for AI systems, modeled on the ATT&CK framework for cybersecurity.
Chris M. Ward, Josh Harguess, Julia Tao +3
Adapts David Bianco's Pyramid of Pain framework to AI security, categorizing AI threats by how difficult they are for adversaries to change.
Apostol Vassilev, Alina Oprea, Alie Fordyce +1
NIST's authoritative taxonomy of adversarial ML attacks and mitigations covering evasion, poisoning, privacy, and abuse attacks against AI systems.
Gary McGraw, Harold Figueroa, Katie McMahon +1
Comprehensive practitioner guide covering AI/ML security from an architectural risk analysis perspective, with practical defense patterns.
OWASP Foundation
Open-source threat modeling tool supporting AI/ML system threat models, data flow diagrams, and STRIDE methodology for GenAI applications.
Xinyue Shen, Zeyuan Chen, Michael Backes +2
Collects and analyzes 6,387 jailbreak prompts from the wild, developing a comprehensive taxonomy of jailbreak techniques and evaluating their effectiveness.
Clark Barrett, Brad Boyd, Elie Burzstein +20
Comprehensive treatment of generative AI security risks across the ML lifecycle with a focus on practical mitigations and deployment considerations.