Prompt validation, sanitization, and input guards
Newest first · 10 reviewed on this page
Search insteadMohammed Kharma, Ahmed Sabbah, Mohammad Alkhanafseh +2
The growing use of Large Language Models (LLMs) for automated code generation has enhanced software development efficiency, but often at the cost of security. Generated code frequently overlooks critical concerns, leaving it vulnerable to issues such as weak encryption and…
Jianwei Tai
We give a quantitative impossibility result for pass-only prompt hardening of code LLMs. For any deterministic prompt filter $h$ and a registered family of finite executable-equivalence task variables $\mathcal Y_{\mathrm{exec}}$, the shared filtered-prompt channel…
Gulshan Saleem, Nisar Ahmed, Muhammad Imran Zaman +1
Prompt injection is ranked as the most critical vulnerability in large language model (LLM) deployments by the OWASP Top 10 for LLM Applications, yet existing defenses operate at isolated pipeline stages and remain incomplete. Input filters cannot inspect retrieved documents,…
Xiaodong Wu, Zhimin Zhao, Qi Li +4
A coding agent edits files and executes shell commands with its developer's privileges, allowing malicious requests to translate directly into harmful actions or functional malware. Existing defenses have complementary limitations: weight-level alignment is unavailable to…
Michal Podpora, Marek Baranowski, Maciej Chopcian +2
Large Language Models with Retrieval-Augmented Generation are considered to be modern, chat-native interfaces to enterprise knowledge. However, deploying such systems safely requires precautions more advanced than input filtering. Numerous LLM-related security threats (including…
Dominik Schwarz
As Large Language Models are increasingly integrated into complex automated pipelines, security vulnerabilities that extend beyond simple input filtering become a practical concern. This work presents a mechanism-centered taxonomy of 41 recurring risk patterns, identified…
Eric Wallace, Kai Xiao, Reimar Leike +3
Proposes an instruction hierarchy for training LLMs to prioritize system prompts over user prompts over third-party content, as a defense against prompt injection.
X. Suo
The critical challenge of prompt injection attacks in Large Language Models (LLMs) integrated applications, a growing concern in the Artificial Intelligence (AI) field. Such attacks, which manipulate LLMs through natural language inputs, pose a significant threat to the security…
Jingwei Yi, Yueqi Xie, Bin Zhu +5
Provides a benchmark for indirect prompt injection attacks and evaluates several defense strategies including perplexity-based detection and sandwich defense.
Yupei Liu, Yuqi Jia, Runpeng Geng +2
Proposes defense mechanisms against prompt injection in LLM systems including isolation-based approaches, input/output filtering, and detection methods.
Seungju Han, Kavel Rao, Allyson Ettinger +5
Open-source moderation tool for detecting safety risks in LLM interactions, trained on a diverse dataset of harmful and benign prompts.
DeadBits
Open-source scanner for detecting prompt injections using vector similarity, YARA rules, text classifiers, and canary tokens.
Guardrails AI
Framework for adding structural, type, and quality guarantees to LLM outputs with validators for PII, toxicity, code security, and factual accuracy.
Protect AI
Comprehensive toolkit for sanitizing LLM prompts and outputs, detecting prompt injection, PII leakage, toxic content, and code vulnerabilities.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi +8
Introduces Llama Guard, an LLM-based safeguard model for classifying safety risks in LLM inputs and outputs, achieving strong performance on standard benchmarks.
Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar +2
Presents NeMo Guardrails, an open-source toolkit for adding programmable safety, security, and privacy rails to LLM-based conversational systems.
Protect AI
Open-source tool designed to detect and prevent prompt injection attacks using multiple detection methods including heuristics, LLM-based analysis, and canary tokens.