Skip to content
Search
paperMay 2026Unreviewed

PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection

Steffen J. Camarato, Yahya Hmaiti, Mandana Ghadamian, David Mohaisen

View resourcePDF

Abstract

Large language models are increasingly used for vulnerability detection, yet their reliability under different prompt formulations remains uncharacterized. We present PromptAudit, a controlled evaluation framework that isolates prompt effects by fixing the dataset, decoding, and parsing while varying only the prompting strategy. Using five prompting strategies across five open-weight models on 1,000 CVEs (6,074 code samples spanning 16 programming languages), we evaluate accuracy, recall, absten

Categories

Benchmarks & Evaluation

Cite

@misc{camarato2026promptaudit,
  title = {{PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection}},
  author = {Steffen J. Camarato and Yahya Hmaiti and Mandana Ghadamian and David Mohaisen},
  year = {2026},
  month = may,
  eprint = {2605.24171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.24171}
}