Determining whether specific data was used in training
Newest first
Search insteadZeyuan Chen, Yihan Ma, Xinyue Shen +2
Large language models (LLMs) show strong performance across many applications, but their ability to memorize and potentially reveal training data raises serious privacy concerns. We introduce the PopQuiz Attack, a black-box membership inference attack that tests whether a model…
Wei Song, Yuxin Cao, Ziqi Ding +3
Video large language models (VideoLLMs) are increasingly trained or instruction-tuned on large-scale video--text corpora collected from heterogeneous sources, raising an immediate privacy question: can an external auditor determine whether a particular video was used during…
Yanming Mu, Hao Hu, Feiyang Li +7
Retrieval-Augmented Generation (RAG) significantly mitigates the hallucinations and domain knowledge deficiency in large language models by incorporating external knowledge bases. However, the multi-module architecture of RAG introduces complex system-level security…
Guilin Deng, Silong Chen, Yuchuan Luo +6
Federated Large Language Models (FedLLMs) enable multiple parties to collaboratively fine-tune LLMs without sharing raw data, addressing challenges of limited resources and privacy concerns. Despite data localization, shared gradients can still expose sensitive information…
He Yang Yuan, Xin Wang, Kundi Yao +3
Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practices can inadvertently expose sensitive information or enable attacks such as log injection, posing…
Ziyi Tong, Feifei Sun, Le Minh Nguyen
Large Language Models (LLMs) have become the predominant paradigm in NLP, advancing both research and industry. As model sizes and pretraining data grow, concerns about Pretraining Data Exposure (PDE) increase due to the scale and opacity of training datasets. PDE refers to…
Ivan Dobrovolskyi
Organizations that scan documents for sensitive information face a practical problem. Cloud services require data to be sent to external infrastructure, while rule-based tools often miss threats that depend on context. This study presents TorchSight, an open-source local system…
Hongjang Yang, Hyunsik Na, Daeseon Choi
LLM-based chatbot agents increasingly process user requests by combining natural-language reasoning with external tools such as web browsing. These capabilities improve usability, but they also create attack surfaces when untrusted external content is processed as part of a…
Youqian Zhang
This work examines an imbalance in artificial intelligence (AI) security research: the field tends to produce more work on attacking AI systems than on defending them. Drawing on related academic papers, we find biased attack-to-defense ratios across subfields, including…
Lubana Isaoglu, Zeynep Orman
The increasing use of Large Language Models (LLMs) has enabled the generation of high-quality synthetic text, providing a potential alternative to sensitive real-world datasets in domains where privacy concerns limit data sharing. However, synthetic text is not inherently…
Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1
Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments. This transition changes the nature of security risk. In agentic settings, failures are no longer…
Hankyul Baek, Jaewon Noh, Sang Seo +5
AI agents are increasingly being adopted in enterprise and personal settings with access to emails, databases, documents, and other tools where they can read, update, and disseminate sensitive information. Much of prior research on data leakage risks in agents has focused on…
Nada Lahjouji, Ashwin Gerard Colaco
Large language model agents increasingly query databases, search document collections, call external APIs, remember past interactions, and act on a user's behalf. As they move from answering questions to operating over sensitive data, privacy becomes harder to enforce. An agent…
SingGuard Team
We present nsfaguard, a guardrail framework for securing agentic AI systems against operational threats, such as prompt injection, sensitive information extraction, malicious code requests, dangerous tool misuse, and resource exhaustion. We first introduce the NSFA taxonomy,…
Akansha Shukla, Emily Bellov, Parth Atulbhai Gandhi +2
Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks. Enforcing required controls consistently is particularly challenging in workflows…
Ethen Santana, Gabriel Gyaase, Hao Zheng
Hardware designs, like software, are susceptible to bugs that can introduce security vulnerabilities and create opportunities for malicious exploitation. Unlike software vulnerabilities, however, hardware flaws become permanently embedded in silicon after fabrication, making…
Longtao Guo, Zelin Zhang, Kaifeng Huang +1
LLM-based web agents automate user tasks by observing webpages and executing browser actions on behalf of users. As these agents operate on real web services, login becomes a sensitive authentication boundary because it involves credentials and sensitive information. Existing…
Xukun Luan, Jinyan Liu, Yuhui Gong +4
Vision-Language models (VLMs) achieve outstanding performance largely due to the amount of training data available on the internet. At the same time, data holders (e.g., artists) urgently need to determine whether their data has been used for model training without…
Minh Tran, Cuong Dang, Tuc Nguyen +10
Retrieval-Augmented Generation (RAG) enhances large language models by grounding outputs in external knowledge, improving factuality and reducing hallucinations. At the same time, the retrieval-augmented pipeline introduces new robustness and security risks, including corpus…
Prince Jha, Samuele Poppi, Nils Lukas
Agentic AI systems are increasingly deployed to process sensitive data at inference time, such as healthcare records or financial documents assembled into a hidden \emph{context} before the system answers. Prior work has studied privacy risks primarily through…
Shengfang Zhai, Leo Marchyok, Yuling Shi +4
Diffusion language models (DLMs) have recently emerged as an alternative modeling paradigm to autoregressive LMs, offering advantages such as parallel generation and bidirectional context modeling. Despite growing interest in their generative capabilities, the privacy risks of…
Kunlin Cai, Kaiyuan Zhang, Zihang Xiang +4
Text-to-Speech (TTS) foundation models are increasingly fine-tuned on private datasets to synthesize highly personalized voices, introducing severe privacy risks by exposing both biometric identities and sensitive speech content. Existing black-box membership inference attacks…
Jiechao Gao, Yuandong Pan, Jie Wang +2
Sensor-intensive environments enable many intelligent services by inferring user applications from heterogeneous data streams. However, not all applications should be exposed: users want some activities to stay private. This creates a tension between inferring applications for…
Ali Akarma, Toqeer Ali Syed, Muhammad Khan +2
As vehicular networks move toward 5G/6G edge intelligence, federated learning (FL) is widely promoted as a privacy-preserving way for vehicles and infrastructure to train shared models without exposing raw sensor data. Yet the updates clients transmit still leak enough…
Jialu Guo, Xiao Han, Junjie Wu
Diffusion models have achieved remarkable success in generative tasks across various areas, however their training process raises significant privacy concerns, particularly under membership inference attacks (MIAs). Prior studies on privacy-preserving of diffusion models fail to…
Shengjie Niu, Yeheng Ge, Jian Huang
Membership inference attacks (MIAs) have emerged as critical tools for auditing privacy risks in large language models (LLMs), aiming to determine whether a given text was included in a model's training corpus. However, most existing MIAs require access to per-token logits or…
Yuhang Fan, Yusi Chen, Kanyu Ye +1
Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead.…
Jiamin Zheng, Hao-Ping Lee, Luo Mai +1
Privacy impact assessment (PIA) is a critical instrument for institutions to proactively identify privacy risks and develop mitigation strategies before system deployment. While mandated across regulatory and institutional contexts, executing PIA requires extensive privacy and…
Qihang Cen, Tianshuo Cong, Da Song +4
Mobile GUI agents increasingly rely on Vision-Language Models (VLMs) to automate smartphone tasks by interpreting screenshot streams. However, this design introduces serious and underexplored privacy risks, including direct leakage of sensitive on-screen information and…
Shuhuai Huang, Jingfeng Zhang, Hong Jia
Harness design has transformed the development of LLM-based agents by integrating memory, tool use, and runtime control. However, this design also introduces security and privacy risks because malicious instructions from external sources may be written into persistent memory and…
Zhe Liu
Large Audio-Language Models (LALMs) have recently shown strong capabilities in speech understanding and question answering (QA), but they also inherit privacy risks from large-scale training data, including the unintended memorization of sensitive information. In this work, we…
Zi-Tai Chen, Reza Shokri
Extracting training data from large language models (LLMs) is a serious privacy breach that exposes (potentially private) data without data owners' consent. Existing extractions follow the generation-then-audit paradigm, where the greedy decoding method in generation limits the…
Dong-Dong Zhao, Jian Chen, Guan-Cheng Lin +3
Code generation benchmarks are widely used to evaluate Large Language Models (LLMs), but benchmark data leakage into training sets can inflate performance and undermine evaluation validity. DetectLeak, a method specifically designed for code generation benchmark leakage…
Ke-Jia Zhang, Tianyuan Zou, Zi-Xuan Gu +1
Applications such as personalized assistance and proprietary document analysis require large language models (LLMs) to generate outputs from private data. Yet powerful LLMs typically cannot be deployed on the resource-constrained devices where private data resides, and uploading…
Yian Wang, Agam Goyal, Eshwar Chandrasekharan +1
Multi-agent LLM systems often coordinate by compressing an upstream interaction into a handoff artifact that downstream agents treat as shared state. We show that this handoff step is a structural source of privacy leakage: summaries preferentially preserve operational facts…
Hao-Cheng Jiang, Hua Shen
As large language models are increasingly adopted in federated learning, protecting user privacy while performing parameter-efficient fine-tuning on distributed private data has become an important challenge. Although clients only share gradients instead of directly uploading…
Jie-Qun Wei, Yuejin Zhang
With the rapid growth of digital information, the risk of sensitive information leakage in textual data, including personally identifiable information, medical privacy, financial data, and corporate confidential information, has become increasingly prominent. Traditional…
Long-Zhu He, Zekun Wen, Chaozhuo Li +1
Large language models (LLMs) have recently advanced graph neural networks (GNNs) by enriching node representations with semantic information, giving rise to LLM-enhanced GNNs that achieve substantial performance gains. However, their vulnerability to privacy attacks, in which…
Hao Chang, Ming-Zhe Chen, Qianqian Zhang
Semantic communication has emerged as a promising paradigm for next-generation wireless systems by transmitting high-level semantic features rather than raw bits. However, collaborative devices and multimodal transmission increase privacy risks because sensitive information may…
Dimitri Staufer, David Hartmann, Ibrahim Baroud
Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential status is uncontrolled, measurements may conflate memorisation, retrieval, name priors and wrong-person attribution. We…
Beining Xu, Hairui Wang, Jiaxin Wang +2
While the privacy risks of multimodal large language models (MLLMs) have drawn significant attention, the unique vulnerabilities of domain-specific MLLMs remain largely underexplored. Focusing on document understanding MLLMs for identity document processing, this paper…
Adam Ait Hsine, A. Arabo
The deployment of large language models (LLMs) in real-world applications introduces a compounding security problem: detecting adversarial inputs such as prompt injection and jailbreak-driven data leakage while simultaneously preventing the detection mechanism itself from…
Wen-Wen Zhao, Jia-Wen Liu
With the rapid popularization of generative artificial intelligence (AI), the risks of misinformation dissemination and user privacy leakage have become key factors influencing users' decisions on the continuous use of AI. Most existing studies focus on the driving mechanisms of…
Nan Yan, Yu-Qing Li, Xiong Wang +5
Federated low-rank adaptation (FedLoRA) allows multiple clients to collaboratively fine-tune large language models (LLMs) on downstream tasks without exposing their private data. To mitigate privacy leakage during aggregation, differential privacy (DP) is widely used to clip and…
Zichun Su, Mi Zhang, Xiaohan Zhang +3
Collaborative inference enables cost-effective deployment of large language models by partitioning layers across multiple participants and forwarding intermediate activations between participants in a pipeline, but these transmitted activations also create a new attack surface:…
Shuaifan Jin, Zhibo Wang, Qiyuan Wang +5
Collaborative inference deploys Large Vision-Language Models (LVLMs) by partitioning computation between edge devices and the cloud. While withholding raw inputs supposedly ensures privacy, transmitting intermediate hidden states exposes a critical attack surface. However, it…
Wei Sun, Xianda Wang, Zhicheng Liang +5
Federated learning (FL) stands out as a promising solution to address the ever-increasing data scarcity problem of large language model (LLM) training through collecting data from distributed sources. The unique challenges however exist in three aspects, the extreme large…
Baiqi Wu, Qing-Ming Li, Chun-Yi Zhou +2
Rapid AI development across industries raises pressing security and privacy risks. This work presents a unified comparison of large language models, AI agents, and embodied agents, introducing a taxonomy of risks spanning data, models, systems, content, and applications,…
Gustavo Viana
Large Language Models (LLMs) are increasingly deployed in production systems, yet their prompt-based interaction paradigm introduces a novel attack surface encompassing prompt injection, instruction hijacking, and sensitive data leakage. This paper proposes and empirically…
Sabin Adhikari, Roshan Paudel, Dipesh Gautam +4
Prompt injection is a serious threat to the security of large language models operating in AI-powered browsers and autonomous web agents, which depend on the ability of those models to interpret instructions correctly as they are used for automated browsing, data extraction or…