Output Filtering and Content Moderation

Objective

Apply layered output filtering and content moderation to LLM and generative AI systems to prevent disclosure of sensitive data, prohibited content, executable payloads, and policy-violating outputs.

Applicability

AI types

LLM, Agentic AI, Generative AI, Multi-modal

Deployment models

SaaS, Self-hosted, Hybrid, Edge

Lifecycle stages

Deployment, Operation & Monitoring

Risk domains

Model, Data

Regulatory regimes

EU AI Act, GDPR, ISO 42001, Banking, Healthcare, Public sector

Company size

SMB, MidMarket, Enterprise

Rationale

Model outputs are an attack surface: prompt injection, training-data extraction, system-prompt leakage, hallucinated PII, code with embedded SSRF/XSS, and policy violations all manifest as output content. OWASP LLM02 (Insecure Output Handling) and LLM06 (Sensitive Information Disclosure) describe the harm pattern. Output controls complement input controls and must be designed as defense in depth.

Control narrative

Every production LLM/generative system applies output controls before content is returned to users, stored, or passed to downstream tools. The output pipeline performs, at minimum: (a) sensitive-data scanning (PII, secrets, regulated identifiers, internal classifiers); (b) prohibited-content moderation (per acceptable-use policy and applicable sectoral rules); (c) system-prompt/IP leakage detection; (d) executable-content handling (HTML/JS escaping for chat surfaces; sandboxed rendering for code; URL allow-listing for outbound links); and (e) hallucination flags (claim verification for high-risk domains). Bypass requires documented business justification and compensating controls. Output events are logged with content fingerprints (not raw content) for incident response.

Test of Design

Procedures

1. Obtain the Output Filtering standard and confirm coverage of sensitive-data, prohibited-content, system-prompt leakage, executable-content handling, and hallucination flagging.
2. Confirm bypass requires documented justification and compensating controls.
3. Confirm logging captures sufficient metadata for incident response without storing raw sensitive content.
4. Confirm integration with the AI System Inventory so every production LLM/generative system is mapped to its output pipeline.
5. Confirm change-management process for output rules (additions, threshold tuning, rollbacks).

Inquiries

• Who owns the output filter rule set?
• How are new sensitive-data patterns added (e.g., new product identifiers)?
• How is false-positive impact monitored (over-blocking that degrades usability)?
• How are bypass requests handled and reviewed?
• How is content moderation handled for non-English languages?

Inspections

• Output Filtering standard.
• Configured output rule sets for sampled systems.
• Bypass request samples with documented justification.
• Output event logs.
• Quarterly tuning reports.

Test of Operating Effectiveness

Procedures

1. From the inventory, sample production LLM/generative systems.
2. For each, inspect the output pipeline configuration and confirm coverage of mandatory categories.
3. Submit a battery of canonical probes (PII patterns, prompt-leakage probes, code-injection probes) in a non-production replica and confirm outputs are blocked or transformed per policy.
4. Sample output event logs and confirm appropriate metadata is captured.
5. Sample bypass approvals and confirm justification and compensating-control documentation.
6. Confirm quarterly tuning occurred and false-positive rates are within tolerance.

Sample-size guidance

Reperformance

• For 1 sampled system, independently exercise 5 PII-extraction probes from a controlled corpus and verify all are blocked or transformed.

Evidence requirements

Required

• Output Filtering standard PDF/Word · At fieldwork
• Output rule-set configurations per sampled system Config export / screenshots · Per sample
• Output event log samples System export · Per sample
• Bypass approval records Workflow export · Per sample

Supporting

• Quarterly tuning reports PDF/Excel · Quarterly

Retention: 1 year for output event metadata; 7 years for incident-related records (regulated)

Framework mappings

Related controls

• AI-CTRL-003

  Adversarial Robustness Testing for LLM Systems

• AI-CTRL-013

  Personal Data Minimization in Training and Inference

• AI-CTRL-019

  Agentic AI Tool Authorization Boundaries

References

• OWASP LLM02:2025 — Insecure Output Handling
• OWASP LLM06:2025 — Sensitive Information Disclosure
• EU AI Act Article 50 — Transparency Obligations