Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

Abstract

This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Secur

Categories

@misc{yohn2026can,
  title = {{Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment}},
  author = {Derek Yohn and Luke Flancher and Mirajul Islam and Khaled Slhoub},
  year = {2026},
  month = jun,
  eprint = {2606.11672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.11672}
}