ATK-008 Output handling
Training Data Extraction via Prompting
Applies to: LLM, Multi-modal · Last reviewed 2026-05-01
Prerequisites
- Target model was trained or fine-tuned on data potentially containing memorized examples (PII, copyrighted text, confidential documents).
- Model is queryable in sufficient volume to attempt extraction.
Test approach
With written authorization and within a documented query budget, the tester applies known training-data-extraction techniques: prompt continuation from short sequences likely to be memorized, divergence prompts, retrieval-style queries targeting known canary content (if such content was inserted at training time), and structured queries inspired by recent academic literature. The tester does not target real personal data of identifiable individuals without separate explicit authorization.
Expected evidence
- Test prompts (or prompt distributions) used.
- Outputs flagged as containing potentially memorized content with similarity scores against known training corpora (where available).
- Query budget consumption.
Detection signals
- High volume of low-entropy completion queries from a single source.
- Repeated requests for continuations of specific short strings.
- Outputs containing PII-like patterns at higher rates than baseline.
Defensive recommendations
- Apply rate limiting and anomaly detection on inference traffic patterns characteristic of extraction.
- Use training-time mitigations (deduplication, differential privacy, canary insertion for monitoring).
- Apply output-side PII detection and redaction.
- Reference AI-CTRL-013 (Personal Data Minimization) for the program-level control.
Framework mappings
- OWASP LLM Top 10: LLM06:2025, LLM10:2025
- MITRE ATLAS: AML.T0024
- NIST AI RMF: MEASURE-2.7, MEASURE-2.10
Related AI Controls Catalog entries
These controls provide the audit test procedures that validate defences against this attack pattern.