Constitutional AI, safety layers, and behavioral constraints
Newest first · 14 reviewed on this page
Search insteadEvan Hubinger, Carson Denison, Jesse Mu +27
Demonstrates that LLMs can be trained with deceptive behaviors (sleeper agents) that persist through standard safety training including RLHF, posing risks for backdoor attacks.
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo +8
Evaluates whether multimodal LLMs aligned to refuse harmful text requests also refuse harmful image-based requests, finding significant gaps.
Eric Wallace, Kai Xiao, Reimar Leike +3
Proposes an instruction hierarchy for training LLMs to prioritize system prompts over user prompts over third-party content, as a defense against prompt injection.
Boyi Wei, Kaixuan Huang, Yangsibo Huang +6
Demonstrates that safety alignment in LLMs is brittle and can be undermined through simple weight pruning or low-rank modifications without any fine-tuning data.
Zhexin Zhang, Leqi Lei, Lindong Wu +2
Large-scale safety evaluation benchmark with 11,435 multiple-choice questions across 7 safety categories in both Chinese and English.
Alexandra Souly, Qingyuan Lu, Dillon Bowen +8
Introduces StrongREJECT, a high-quality evaluation benchmark for measuring how well LLMs refuse harmful requests.
Tianrong Zhang, Bochuan Cao, Yuanpu Cao +3
The recent breakthrough in large language models (LLMs) such as ChatGPT has revolutionized production processes at an unprecedented pace. Alongside this progress also comes mounting concerns about LLMs' susceptibility to jailbreaking attacks, which leads to the generation of…
Shuo Chen, Zhen Han, Bailan He +5
Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the textual modality and extend the jailbreak attack to Multimodal Large Language Models (MLLMs) by…
Seungju Han, Kavel Rao, Allyson Ettinger +5
Open-source moderation tool for detecting safety risks in LLM interactions, trained on a diverse dataset of harmful and benign prompts.
Oleksandr Muliarevych
This article examines cybersecurity vulnerabilities in systems utilizing Language Model Interfaces, focusing on the challenges of building secure systems. It provides an overview of current interfaces and their associated risks. A key contribution is the design of a prompt…
Guardrails AI
Framework for adding structural, type, and quality guarantees to LLM outputs with validators for PII, toxicity, code security, and factual accuracy.
Protect AI
Comprehensive toolkit for sanitizing LLM prompts and outputs, detecting prompt injection, PII leakage, toxic content, and code vulnerabilities.
Erfan Shayegani, Yue Dong, Nael B. Abu-Ghazaleh
We introduce new jailbreak attacks on vision language models (VLMs), which use aligned LLMs and are resilient to text-only jailbreak attacks. Specifically, we develop cross-modality attacks on alignment where we pair adversarial images going through the vision encoder with…
Hakan Inan, Kartikeya Upasani, Jianfeng Chi +8
Introduces Llama Guard, an LLM-based safeguard model for classifying safety risks in LLM inputs and outputs, achieving strong performance on standard benchmarks.
Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar +2
Presents NeMo Guardrails, an open-source toolkit for adding programmable safety, security, and privacy rails to LLM-based conversational systems.
Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish
Shows that LoRA fine-tuning with as few as 100 examples can remove safety guardrails from Llama 2-Chat, raising concerns about fine-tuning access to aligned models.
Xianjun Yang, Xiao Wang, Qi Zhang +4
Shows that RLHF can introduce shadow alignment where models exhibit harmful behaviors not present in the base model.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu +44
Introduces Constitutional AI (CAI), a method for training AI systems to be harmless using a set of principles (a constitution) and AI-generated feedback, reducing reliance on human red teamers.