Training data, fine-tuning, and RAG poisoning attacks
Newest first · 6 reviewed on this page
Search insteadRizwan Tanveer
Background. Adversarial machine learning has progressed from a marginal concern within machine learning research into a first-order discipline for the secure deployment of artificial intelligence systems in regulated and operational environments. The contemporary threat…
Xiaoyun Xu
This thesis addresses the critical challenge of adversarial machine learning in deep learning models, focusing on the defense mechanisms against evasion (adversarial) attacks and backdoor attacks. Part I analyzes evasion attacks through the lens of information bottleneck theory,…
Shruti Bamboria, Kiran R. Dodiya, Kapil Kumar
This chapter examines the intersection of quantum computing, cybersecurity, and adversarial machine learning (AML), outlining both transformative opportunities and emerging risks. Quantum computing, leveraging superposition and entanglement, offers unparalleled computational…
Jinyu Chen, Jixiao Yang, Ziyang Zeng +3
Large Language Model (LLM)-based multi-agent systems have demonstrated remarkable capabilities across diverse applications, yet they face critical security challenges including backdoor attacks, prompt injection, and privacy leakage. Existing defense mechanisms typically address…
Eranga Bandara, Amin Hass, Ross Gore +8
AI agent-based systems are becoming increasingly integral to modern software architectures, enabling autonomous decision-making, dynamic task execution, and multimodal interactions through large language models (LLMs). However, these systems introduce novel and evolving security…
Evan Hubinger, Carson Denison, Jesse Mu +27
Demonstrates that LLMs can be trained with deceptive behaviors (sleeper agents) that persist through standard safety training including RLHF, posing risks for backdoor attacks.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo +6
Demonstrates practical attacks to poison web-scale datasets like LAION by purchasing expired domains, affecting 0.01% of a dataset for under $60.
Wei Zou, Runpeng Geng, Binghui Wang +1
Demonstrates knowledge poisoning attacks against RAG systems where adversaries inject malicious texts into the knowledge database to manipulate LLM outputs.
Jiaqi Xue, Mengxin Zheng, Ting Hua +4
Proposes TrojLLM, a black-box attack that generates universal trojan prompts to compromise LLMs without access to model internals.
Zhen Xiang, Fengqing Jiang, Zidi Xiong +3
Demonstrates backdoor attacks on chain-of-thought reasoning in LLMs where poisoned demonstrations cause incorrect reasoning chains.
Alexander Wan, Eric Wallace, Sheng Shen +1
Shows that adversaries can insert poisoned examples into instruction-tuning datasets, causing models to generate targeted outputs for attacker-chosen triggers.