Guardrail bypass and alignment subversion techniques
Newest first · 13 reviewed on this page
Search insteadXiangyu Qi, Kaixuan Huang, Ashwinee Panda +3
Shows adversarial images can jailbreak multimodal LLMs that are robust to text-only attacks, bypassing alignment through the visual channel.
Mantas Mazeika, Long Phan, Xuwang Yin +9
Introduces HarmBench, a standardized framework for evaluating automated red teaming methods and robust refusal in LLMs with a comprehensive behavior taxonomy.
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo +8
Evaluates whether multimodal LLMs aligned to refuse harmful text requests also refuse harmful image-based requests, finding significant gaps.
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik +4
Introduces TAP using an LLM to iteratively refine jailbreak prompts against black-box target models with high success rates.
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang +4
Demonstrates that LLMs can be jailbroken using cipher-based encoding, bypassing safety training designed for natural language.
Xiangming Gu, Xiaosen Zheng, Tianyu Pang +5
A multimodal large language model (MLLM) agent can receive instructions, capture images, retrieve histories from memory, and decide which tools to use. Nonetheless, red-teaming efforts have revealed that adversarial images/prompts can jailbreak an MLLM and cause unaligned…
Leo Schwinn, David Dobre, Stephan Gunnemann +1
Systematizes adversarial attacks and defenses for LLMs, connecting them to the classical adversarial ML literature while identifying LLM-specific threats.
Yi Zeng, Hongpeng Lin, Jingwen Zhang +3
Applies social science persuasion techniques to jailbreak LLMs, showing high attack success rates using persuasion taxonomy.
Zhiyuan Chang, Mingyang Li, Yi Liu +3
With the development of LLMs, the security threats of LLMs are getting more and more attention. Numerous jailbreak attacks have been proposed to assess the security defense of LLMs. Current jailbreak attacks primarily utilize scenario camouflage techniques. However their…
Yujun Zhou, Yufei Han, Haomin Zhuang +5
Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications. However, concerns regarding their security, particularly the vulnerability to jailbreak attacks, persist. Drawing inspiration from adversarial training in deep learning and LLM agent…
Tianrong Zhang, Bochuan Cao, Yuanpu Cao +3
The recent breakthrough in large language models (LLMs) such as ChatGPT has revolutionized production processes at an unprecedented pace. Alongside this progress also comes mounting concerns about LLMs' susceptibility to jailbreaking attacks, which leads to the generation of…
Fan Liu, Zhao Xu, Hao Liu
Although safely enhanced Large Language Models (LLMs) have achieved remarkable success in tackling various complex tasks in a zero-shot manner, they remain susceptible to jailbreak attacks, particularly the unknown jailbreak attack. To enhance LLMs' generalized defense…
Shuo Chen, Zhen Han, Bailan He +5
Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the textual modality and extend the jailbreak attack to Multimodal Large Language Models (MLLMs) by…
Bhagyajit Pingua, Deepak Murmu, Meenakshi Kandpal +4
Large language models (LLMs) have become transformative tools in areas like text generation, natural language processing, and conversational AI. However, their widespread use introduces security risks, such as jailbreak attacks, which exploit LLM’s vulnerabilities to manipulate…
Miao Yu, Junfeng Fang, Yingjie Zhou +4
While safety-aligned large language models (LLMs) are increasingly used as the cornerstone for powerful systems such as multi-agent frameworks to solve complex real-world problems, they still suffer from potential adversarial queries, such as jailbreak attacks, which attempt to…
Zhihua Jin, Shiyi Liu, Haotian Li +2
Large Language Models (LLMs) have gained significant attention but also raised concerns due to the risk of misuse. Jailbreak prompts, a popular type of adversarial attack towards LLMs, have appeared and constantly evolved to breach the safety protocols of LLMs. To address this…
Anthropic
Reveals many-shot jailbreaking, a technique exploiting long context windows by including many examples of harmful Q&A pairs to override safety training.
Andy Zou, Zifan Wang, Nicholas Carlini +3
Proposes an automated method (GCG) to generate adversarial suffixes that cause aligned LLMs to produce harmful content, with attacks transferring across models including ChatGPT and Claude.
Xinyue Shen, Zeyuan Chen, Michael Backes +2
Collects and analyzes 6,387 jailbreak prompts from the wild, developing a comprehensive taxonomy of jailbreak techniques and evaluating their effectiveness.
Erfan Shayegani, Yue Dong, Nael B. Abu-Ghazaleh
We introduce new jailbreak attacks on vision language models (VLMs), which use aligned LLMs and are resilient to text-only jailbreak attacks. Specifically, we develop cross-modality attacks on alignment where we pair adversarial images going through the vision encoder with…
Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish
Shows that LoRA fine-tuning with as few as 100 examples can remove safety guardrails from Llama 2-Chat, raising concerns about fine-tuning access to aligned models.
Sander Schulhoff, Jeremy Pinto, Anaum Khan +7
Presents results from a global prompt hacking competition with 600K+ adversarial prompts, revealing systemic LLM vulnerabilities across multiple models and defense strategies.
Haoran Li, Dadi Guo, Wei Fan +4
Demonstrates multi-step jailbreaking attacks to extract personal information from ChatGPT, showing how sequential prompting can bypass safety measures.