Skip to content
Search
paper2024ReviewedOpen access

Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models

Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov

arXiv preprint

View resourcePDF

Abstract

Introduces CyberSecEval, a benchmark for evaluating the cybersecurity risks of LLM code generation, including insecure code suggestions.

Categories

Benchmarks & EvaluationSupply Chain AttacksRed Teaming
#CyberSecEval#code-security#Meta#benchmark

Framework mappings

OWASP Top 10 for LLM Applications
  • LLM05Improper Output Handling
NIST AI Risk Management Framework
  • MEASUREMeasure

Cite

@misc{bhatt2024purple,
  title = {{Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models}},
  author = {Manish Bhatt and Sahana Chennabasappa and Cyrus Nikolaidis and Shengye Wan and Ivan Evtimov},
  year = {2024},
  eprint = {2312.04724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.04724}
}