paperMay 2026Unreviewed

Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)

Richard J. Young, Gregory D. Moody

Abstract

The evaluation of large language model refusal on malicious-coding tasks now spans at least thirteen publicly released prompt corpora (AdvBench, the CyberSecEval family, RMCBench, RedCode, MCGMark, JailbreakBench, CySecBench, MalwareBench, CIRCLE, MOCHA, ASTRA, Scam2Prompt / Innoc2Scam-bench, and JAWS-Bench), each constructed under a different protocol, released under different licensing terms, and validated (or not) against different inter-rater reliability standards. Existing surveys treat cod

Framework mappings

OWASP Top 10 for LLM Applications

LLM01Prompt Injection

MITRE ATLAS

AML.T0054LLM Jailbreak

Suggested from the entry's categories.

Cite

@misc{young2026refusal,
  title = {{Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)}},
  author = {Richard J. Young and Gregory D. Moody},
  year = {2026},
  month = may,
  eprint = {2605.20351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.20351}
}