Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray, Alexey A. Shvets

Abstract

We introduce a reusable framework for auditing whether LLM attack benchmarks collectively cover the threat surface: a 4$\times$6 Target $\times$ Technique matrix grounded in STRIDE, constructed from a 507-leaf taxonomy -- 401 data-populated and 106 threat-model-derived leaves -- of inference-time attacks extracted from 932 arXiv security studies (2023--2026). The matrix enables benchmark-external validation -- auditing collective coverage rather than individual benchmark consistency. Applying it

Categories

Cite This Resource

@article{llmsec202600063,
  title = {Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks},
  author = {Karthik Raghu Iyer and Yazdan Jamshidi and Nicholas Bray and Alexey A. Shvets},
  year = {2026},
  url = {https://arxiv.org/abs/2605.15118},
}