Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

Abstract

Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserves context-aware vulnerable--patched pairs, and introduces two diagnostic metrics: the Directional F

Categories

@misc{zibaeirad2026calibration,
  title = {{Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software}},
  author = {Arastoo Zibaeirad and Marco Vieira},
  year = {2026},
  month = jun,
  eprint = {2606.20502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.20502}
}