Adversarial Robustness Testing for LLM Systems

Objective

Validate that production-bound LLM and agentic AI systems have been tested against direct prompt injection, indirect prompt injection, jailbreak, refusal evasion, and (where applicable) multi-modal adversarial inputs, with documented findings, remediation, and re-test cycles.

Applicability

AI types

LLM, Agentic AI, Generative AI, Multi-modal

Deployment models

SaaS, Self-hosted, Hybrid, Edge

Lifecycle stages

Evaluation & Testing, Operation & Monitoring

Risk domains

Model, Data

Regulatory regimes

EU AI Act, ISO 42001, NIST AI RMF, Banking, Healthcare

Company size

MidMarket, Enterprise

Rationale

Prompt injection is the #1 OWASP risk for LLM applications and the most reliable foothold for adversaries against AI systems. Without periodic, documented adversarial testing, organizations cannot demonstrate evaluative due diligence required by ISO/IEC 42001, the EU AI Act (Article 15 — accuracy, robustness, cybersecurity), and NIST AI RMF MEASURE-2.7. Adversarial testing also produces the evidence base for risk-acceptance decisions when residual issues cannot be eliminated.

Control narrative

Each LLM and agentic AI system within scope undergoes adversarial testing at defined milestones: pre-production (mandatory before approval to deploy), upon any material change (new model version, expanded toolset, new data sources), and at minimum quarterly thereafter for systems in production. Testing covers, at minimum: direct prompt injection across user-controllable inputs; indirect prompt injection through RAG sources, document uploads, tool outputs, and (where applicable) multi-modal channels; refusal evasion / jailbreak techniques; system prompt leakage; and output handling weaknesses. Tests are executed by an internal AI Red Team, qualified third party, or a documented combination. Findings are recorded with severity, evidence, root cause, remediation owner, target date, and verification of re-test. The AI Governance lead is accountable for the catalog of tested systems and the open-findings register.

Test of Design

Procedures

1. Obtain the Adversarial Testing standard and confirm coverage of: direct prompt injection, indirect prompt injection, refusal evasion, system prompt leakage, output handling, and multi-modal channels where applicable.
2. Confirm testing milestones are documented: pre-production, material change, periodic (≤ quarterly for production systems).
3. Confirm the standard requires testing by qualified parties (internal AI Red Team or qualified third party) with defined competency requirements.
4. Confirm a findings register exists with required fields: severity, evidence, root cause, remediation owner, target date, re-test verification.
5. Confirm linkage to the AI System Inventory (AI-CTRL-001) so every in-scope system maps to a testing record.
6. Confirm the standard references an AI Red Team engagement model and Rules of Engagement.

Inquiries

• Who is qualified to perform adversarial testing? How is competency assessed?
• How are third-party testers selected and overseen?
• How is testing scope kept current with evolving threat patterns (e.g., new OWASP LLM Top 10 entries, MITRE ATLAS techniques)?
• What is the escalation path when testing reveals critical findings?
• How is testing scope adjusted for agentic systems (tool authorization, multi-agent collusion)?

Inspections

• Adversarial Testing standard / policy.
• Approved testing methodology document.
• Findings register schema and a sample of records.
• Third-party tester agreements where applicable.
• Mapping of AI systems in the inventory to their testing records.

Test of Operating Effectiveness

Procedures

1. From the AI System Inventory, select a sample of LLM and agentic AI systems in production during the audit period.
2. For each sampled system, obtain testing records and confirm: pre-production test occurred prior to deployment; material-change tests are present for each material change in the audit period; periodic tests occurred at minimum quarterly.
3. Inspect the test reports and confirm coverage of mandatory categories per the standard (direct PI, indirect PI, refusal evasion, prompt leakage, output handling, multi-modal where applicable).
4. For findings raised during the audit period, confirm: severity is assigned consistent with the standard; remediation owner and target date are recorded; re-test is verified for closed items; open items past target are tracked and escalated.
5. Independently re-test a sample of closed high or critical findings by reviewing the re-test evidence (test artifacts, screenshots, logs).
6. For systems tested by third parties, confirm tester qualifications were documented and contracts include scope, rules of engagement, and non-disclosure terms.
7. Confirm the catalog of tested systems reconciles to the AI System Inventory with no untested production LLM/agentic systems.

Sample-size guidance

Reperformance

• For 2 closed high or critical findings, independently corroborate the re-test by inspecting raw test artifacts (request/response logs, test harness output).
• For 1 system, independently execute 5 representative prompt-injection probes from public corpora (OWASP, MITRE ATLAS) against a non-production replica and compare results to the latest test report.

Evidence requirements

Required

• Adversarial Testing standard / policy PDF/Word · At fieldwork
• Findings register export CSV/Excel · At fieldwork
• Test reports for sampled systems PDF/Word · Per sample
• Re-test evidence for closed high/critical findings Logs/screenshots/PDF · Per sample
• Third-party tester agreements (where applicable) PDF · Per sample

Supporting

• Tester competency records HR / training records · Per sample
• Methodology versioning / changelog Document or repo · Annual

Retention: 7 years for regulated environments; 3 years otherwise

Framework mappings

Related controls

• AI-CTRL-001

  AI System Inventory and Classification

• AI-CTRL-007

  Output Filtering and Content Moderation

• AI-CTRL-014

  Adversarial Robustness Testing for Predictive ML and Computer Vision

References

• OWASP Top 10 for LLM Applications 2025
• MITRE ATLAS
• NIST AI RMF — MEASURE-2.7 Adversarial Testing
• EU AI Act Article 15 — Accuracy, robustness and cybersecurity