Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

Abstract

Everything a language model sees is tokens. The serving stack knows what each span is -- user input, tool output, instructions -- but the model must keep track of that itself, and it can lose track or be confused: text can be written to read like anything. Prompt injection is a natural exploit of this phenomenon. By scrambling the model's understanding of span identity, an attacker can induce unwanted and potentially dangerous actions. Adding a non-textual channel to the model's input -- a way t

Categories

Framework mappings

Suggested from the entry's categories.

@misc{penman2026semantic,
  title = {{Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors}},
  author = {Joshua Penman},
  year = {2026},
  month = aug,
  eprint = {2608.23873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.23873}
}