Skip to content
Search
paperApril 2026Unreviewed

RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code

John Pellew, Faizan Raza

View resourcePDF

Abstract

How do security scanners perform on real-world code? We present RealVuln, the first open-source benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on 26 intentionally vulnerable Python repositories (educational and Capture-The-Flag applications) with 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps). We test 15 scanners (3 Rule-Based SAST, 10 General-Purpose LLM, 2 Security-Specialized) and rank them by F3 score (beta=3, weighting

Categories

Benchmarks & Evaluation

Cite

@misc{pellew2026realvuln,
  title = {{RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code}},
  author = {John Pellew and Faizan Raza},
  year = {2026},
  month = apr,
  eprint = {2604.13764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.13764}
}