AI red team methodology, automation, and frameworks
Newest first · 11 reviewed on this page
Search insteadMadhava Gaikwad
Recent studies report that automated red-teaming finds more vulnerabilities, at lower cost, than human red-teaming on standard AI safety benchmarks, and some read this as evidence that human evaluators are becoming dispensable. The comparison measures one thing and the…
Yi Ting Shen, Kentaroh Toyoda, Alex Leung
Automated red-team attacks and blue-team defenses for large language models (LLMs) are advancing quickly. However, attackers and defenders are built and tested in isolation, and the resulting scores are hard to trust. To tackle this, we present ACEA (Adversarial Co-Evolution…
Tejasvi C. Addagada
Aligned language models fail under two independent pressures: the structural jailbreak class recently formalized as Involuntary In-Context Learning (IICL), which reframes a harmful request as the final missing cell of a data-labeling task completed by pattern rather than judged…
Doyun Kim, Chanwoo Kim, Sugyeong Eo +2
LLM-based agent systems increasingly adopt skill-based architectures to reduce repetitive reasoning costs and improve stable, efficient task execution. Recent studies propose self-evolving agents that autonomously generate, refine, and reuse skills from past experiences to…
Shashwat Pandey, Satwik Pandey, S. Raghu
Aligning deployed language models requires knowing when their outputs can be trusted, yet on-device models now ship to hundreds of millions of devices with no server-side moderation, and the configuration developers can actually deploy is rarely audited independently. We present…
Qiuxiang Li, Ke Xu, Yubin Qu +1
Tong Zhang, M. Alfarra, Carlos Hinojosa +2
As text-to-image generative models advance, they raise critical safety concerns, particularly the generation of Not-Safe-For-Work (NSFW) content such as violence and nudity, further exacerbated by red-teaming adversarial attacks. Existing defenses predominantly operate under…
Zheng Lin, Yuzhen Huang, Zhenxing Niu +2
Long-term memory enables LLM agents to retain useful information across sessions, but also creates an attack surface through which adversaries may poison an agent's persistent memory to steer its behavior. Existing memory poisoning attacks mainly rely on individually malicious…
Anita Ding, Anthony J. Rose, Mark G. Reith
Active Directory (AD) underpins enterprise identity management and is among the most heavily targeted assets in modern enterprise networks. When red teams emulate attackers targeting it, they typically chain BloodHound for relationship mapping, Impacket for protocol…
Regaan R
The rapid deployment of large language models (LLMs) in production environments has introduced a new class of security vulnerabilities that traditional software testing methodologies are ill-equipped to address. I present Basilisk, an opensource AI red-teaming framework that…
Wesley Gao
Tool-using language-model agents can convert indirect prompt injection into consequential actions, making guardrail quality a joint security, utility, and efficiency problem. This study evaluates a ReAct-style control, native tool filtering, deterministic self-verification, a…
Patrick Chao, Alexander Robey, Edgar Dobriban +3
Uses an attacker LLM to automatically generate jailbreak prompts through iterative refinement achieving high success with only black-box access.
L. Ahmad, S. Agarwal, Michael Lampe +1
Red teaming has emerged as a critical practice in assessing the possible risks of AI models and systems. It aids in the discovery of novel risks, stress testing possible gaps in existing mitigations, enriching existing quantitative safety metrics, facilitating the creation of…
Chetan Pathade
Large Language Models (LLMs) are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that override alignment safeguards. This paper provides a…
Shiyue Zhang, M. Dredze, AI Bloomberg +77
Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming. However, despite the widespread use of the Retrieval-Augmented Generation (RAG) framework, AI safety work focuses on standard LLMs, which means we know little…
Francisco Eiras, Eliott Zemour, Eric Lin +1
Large Language Model (LLM) based judges form the underpinnings of key safety evaluation processes such as offline benchmarking, automated red-teaming, and online guardrailing. This widespread requirement raises the crucial question: can we trust the evaluations of these…
Ian R. McKenzie, O. Hollinsworth, Tom Tseng +5
Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic guards their latest Claude 4 Opus model using one such defense pipeline, and other frontier developers including Google DeepMind and OpenAI pledge to soon…
R. Draelos, Samina Afreen, Barbara Blasko +14
Millions of patients are regularly using large language model (LLM) chatbots for medical advice, raising patient safety concerns. This physician-led red-teaming study compares the safety of four publicly available chatbots—Claude by Anthropic, Gemini by Google, GPT-4o by OpenAI,…
Wesley Hanwen Deng, Sunnie S. Y. Kim, Akshita Jha +4
Recent developments in AI governance and safety research have called for red-teaming methods that can effectively surface potential risks posed by AI models. Many of these calls have emphasized how the identities and backgrounds of red-teamers can shape their red-teaming…
Ren-Jian Wang, Ke Xue, Zeyu Qin +5
Ensuring safety of large language models (LLMs) is important. Red teaming--a systematic approach to identifying adversarial prompts that elicit harmful responses from target LLMs--has emerged as a crucial safety evaluation method. Within this framework, the diversity of…
Jigang Fan, Zhenghong Zhou, Ruofan Jin +3
Proteins play crucial roles in almost all biological processes. The advancement of deep learning has greatly accelerated the development of protein foundation models, leading to significant successes in protein understanding and design. However, the lack of systematic…
A. Asad, Stephen Obadinma, Radin Shayanfar +1
We introduce RedDebate, a novel multi-agent debate framework that provides the foundation for Large Language Models (LLMs) to identify and mitigate their unsafe behaviours. Existing AI safety approaches often rely on costly human evaluation or isolated single-model assessment,…
Ruixuan Huang, Xunguang Wang, Zongjie Li +2
Despite the growing interest in jailbreak methods as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed evaluation system designs have led to significant discrepancies in their effectiveness assessments. We conduct a systematic…
Maissa Trabilsy, Srinivagasam Prabha, C. A. Gomez-Cabello +7
The increasing integration of large language models (LLMs) into healthcare presents significant opportunities, but also critical risks related to patient safety, accuracy, and ethical alignment. Despite these concerns, no standardized framework exists for systematically…
Kai Zhou, Ahmed Elgohary, S. M. Iftekhar +80
The ability of LLM agents to plan and invoke tools exposes them to new safety risks, making a comprehensive red-teaming system crucial for discovering vulnerabilities and ensuring their safe deployment. We present SIRAJ: a generic red-teaming framework for arbitrary black-box…
Liming Lu, Xiang Gu, Junyu Huang +5
Large Language Models (LLMs) are increasingly used in agentic systems, where their interactions with diverse tools and environments create complex, multi-stage safety challenges. However, existing benchmarks mostly rely on static, single-turn assessments that miss…
Blake Bullwinkel, Amanda Minnich, Shiven Chawla +23
Shares lessons from Microsoft's AI red team operations including methodology, tooling, common failure modes, and best practices.
Kai Hu, Abhinav Aggarwal, Mehran Khodabandeh +6
This paper introduces Jailbreak-Zero, a novel red teaming methodology that shifts the paradigm of Large Language Model (LLM) safety evaluation from a constrained example-based approach to a more expansive and effective policy-based framework. By leveraging an attack LLM to…
Syed-Amad Hussain, Daniel I. Jackson, Ashley Lewis +2
Introduction: Health-related chatbots are increasingly used to mediate conversations that carry clinical significance and emotional weight. Retrieval-augmented generation (RAG) can reduce factual errors ('hallucinations'), but the risks remain, with additional challenges coming…
Xiaogeng Liu, Nan Xu, Muhao Chen +1
Proposes AutoDAN, a method for automatically generating stealthy jailbreak prompts that are semantically meaningful and can bypass perplexity-based defenses.
Mantas Mazeika, Long Phan, Xuwang Yin +9
Introduces HarmBench, a standardized framework for evaluating automated red teaming methods and robust refusal in LLMs with a comprehensive behavior taxonomy.
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik +4
Introduces TAP using an LLM to iteratively refine jailbreak prompts against black-box target models with high success rates.
Michael Feffer, Anusha Sinha, Zachary Chase Lipton +1
In response to rising concerns surrounding the safety, security, and trustworthiness of Generative AI (GenAI) models, practitioners and regulators alike have pointed to AI red-teaming as a key component of their strategies for identifying and mitigating these risks. However,…
Xiangming Gu, Xiaosen Zheng, Tianyu Pang +5
A multimodal large language model (MLLM) agent can receive instructions, capture images, retrieve histories from memory, and decide which tools to use. Nonetheless, red-teaming efforts have revealed that adversarial images/prompts can jailbreak an MLLM and cause unaligned…
Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis +2
Introduces CyberSecEval, a benchmark for evaluating the cybersecurity risks of LLM code generation, including insecure code suggestions.
Shayne Longpre, Sayash Kapoor, Kevin Klyman +20
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by prominent AI companies to deter model misuse have disincentives on good faith safety evaluations. This…
Alexandra Souly, Qingyuan Lu, Dillon Bowen +8
Introduces StrongREJECT, a high-quality evaluation benchmark for measuring how well LLMs refuse harmful requests.
Leon Derczynski, Erick Galinkin, Jeffrey Martin +2
Presents garak, an open-source framework for systematically probing LLM vulnerabilities including prompt injection, data leakage, and toxicity generation.
Shuo Chen, Zhen Han, Bailan He +5
Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the textual modality and extend the jailbreak attack to Multimodal Large Language Models (MLLMs) by…
Microsoft AI Red Team
Microsoft's open-source framework for red teaming generative AI systems, supporting automated prompt generation, attack strategies, and scoring of AI responses.
Bhaktipriya Radharapu, Kevin Robinson, L. Aroyo +1
Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it…
Alex Mei, Sharon Levy, W. Wang
As large language models are integrated into society, robustness toward a suite of prompts is increasingly important to maintain reliability in a high-variance environment.Robustness evaluations must comprehensively encapsulate the various settings in which a user may invoke an…
OpenAI
OpenAI's approach to tracking, evaluating, forecasting, and protecting against catastrophic risks of frontier AI models.
Deep Ganguli, Liane Lovitt, Jackson Kernion +30
Describes Anthropic's early red teaming methodology for language models, documenting methods, scaling behaviors, and lessons for identifying harmful outputs.