AIRed Team Framework GitHub
← All attack patterns
ATK-001 Prompt manipulation

Direct Prompt Injection — User Input

Applies to: LLM, RAG, Agentic AI, Multi-modal · Last reviewed 2026-05-01

Prerequisites

  • Target system accepts user-controllable text input that is concatenated with system instructions in the model prompt.
  • No robust prompt-structure enforcement separates trusted system instructions from untrusted user content.

Test approach

With written authorization, the tester submits user inputs designed to override or supplement the system prompt — for example, instructions framed as if from a system role, instructions in different languages, instructions hidden in code blocks or markdown, and instructions cascaded across multi-turn conversations. Test inputs aim to elicit: disclosure of the system prompt; production of policy-violating output; invocation of tools the user is not authorized to invoke; alteration of downstream agent behavior. The tester records which prompts succeed at which boundaries.

Expected evidence

  • Full request/response logs for each test prompt.
  • Model output demonstrating the injected instruction was followed.
  • System prompt content if disclosed.
  • Tool-invocation logs (where applicable) showing unauthorized tool use triggered by injection.

Detection signals

  • Outputs containing artifacts characteristic of the system prompt (length, structure, phrasing).
  • Anomalous tool invocations not consistent with the user's stated intent.
  • Inputs with high token-density of imperative verbs or system-style framing.
  • Inputs in unexpected languages or with unusual encoding (base64, ROT13, leetspeak) used to evade guardrails.

Defensive recommendations

  • Adopt structured prompting where the model API supports trust-tagged role delimiters.
  • Enforce input-side filtering for known injection patterns; tune to minimize over-blocking.
  • Constrain tool authorization (least privilege) so even successful injection cannot reach high-impact tools without independent authorization.
  • Include direct PI test cases in the pre-production evaluation gate and in periodic re-tests.
  • Cross-reference AI-CTRL-003 (Adversarial Robustness Testing for LLM Systems) and AI-CTRL-007 (Output Filtering).

Framework mappings

  • OWASP LLM Top 10: LLM01:2025
  • OWASP Agentic Top 10: AAI01
  • MITRE ATLAS: AML.T0051
  • NIST AI RMF: MEASURE-2.7

Related AI Controls Catalog entries

These controls provide the audit test procedures that validate defences against this attack pattern.

References