Content filtering, PII redaction, and output safety
Newest first · 5 reviewed on this page
Search insteadXiangyu Yin, Yi Qi, Chih-Hong Cheng
Retrieval-Augmented Generation (RAG) improves the reliability of large language model applications by grounding generation in retrieved evidence, but it also introduces a new attack surface: corpus poisoning. In this setting, an adversary injects or edits passages so that they…
Leonhard Waibl, Felix Michalak, Hadrien Mariaccia
LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails…
Seth Dobrin, Łukasz Chmiel
AI agents are granted access to tools, APIs, and other infrastructure, making them active principals in those systems. The dominant approach places controls inside the agent's own runtime: system prompts, output filters, and guardrail libraries. Any control in the agent's…
Albus W. Ng, Yi Han, Jusheng Zhang +1
The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety…
Bhavesh B. Prajapati, Bhavya Shah
Large language model (LLM) applications now combine hidden prompts, retrieval pipelines, memory stores, content filters, tool calls, delegated identities, and downstream automation. Security reviewers are increasingly asked to assess such systems without access to source code,…
Dominik Schwarz
The security of Large Language Model (LLM) applications is fundamentally challenged by "formfirst" attacks like prompt injection and jailbreaking, where malicious instructions are embedded within user inputs. Conventional defenses, which rely on post hoc output filtering, are…
Seungju Han, Kavel Rao, Allyson Ettinger +5
Open-source moderation tool for detecting safety risks in LLM interactions, trained on a diverse dataset of harmful and benign prompts.
Guardrails AI
Framework for adding structural, type, and quality guarantees to LLM outputs with validators for PII, toxicity, code security, and factual accuracy.
Protect AI
Comprehensive toolkit for sanitizing LLM prompts and outputs, detecting prompt injection, PII leakage, toxic content, and code vulnerabilities.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi +8
Introduces Llama Guard, an LLM-based safeguard model for classifying safety risks in LLM inputs and outputs, achieving strong performance on standard benchmarks.
Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar +2
Presents NeMo Guardrails, an open-source toolkit for adding programmable safety, security, and privacy rails to LLM-based conversational systems.