paperMay 2026Unreviewed

Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray, Alexey A. Shvets

Abstract

We introduce a reusable framework for auditing whether LLM attack benchmarks collectively cover the threat surface: a 4$\times$6 Target $\times$ Technique matrix grounded in STRIDE, constructed from a 507-leaf taxonomy -- 401 data-populated and 106 threat-model-derived leaves -- of inference-time attacks extracted from 932 arXiv security studies (2023--2026). The matrix enables benchmark-external validation -- auditing collective coverage rather than individual benchmark consistency. Applying it

Cite

@misc{iyer2026talk,
  title = {{Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks}},
  author = {Karthik Raghu Iyer and Yazdan Jamshidi and Nicholas Bray and Alexey A. Shvets},
  year = {2026},
  month = may,
  eprint = {2605.15118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.15118}
}