paperJune 2026Unreviewed

AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs

Mohamadreza Rostami, Nikhilesh Singh, Stephen Muttathil, Lichao Wu, Chen Chen, Huimin Li, Jeyavijayan Rajendran, Ahmad-Reza Sadeghi

View resource PDF

Abstract

Hardware security competitions such as HackTheSilicon serve as benchmarking platforms for evaluating vulnerability detection methods and for training humans and AI. However, our study reveals that LLMs threaten their validity. Instead of genuine security reasoning, detectors exploit a diff-style syntactic comparison, achieving an 83% detection rate, undermining fair evaluation. To mitigate this, we propose the first LLM-oriented, semantics-preserving obfuscation framework for these benchmarks. U

Cite

@misc{rostami2026attackonctf,
  title = {{AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs}},
  author = {Mohamadreza Rostami and Nikhilesh Singh and Stephen Muttathil and Lichao Wu and Chen Chen and Huimin Li and Jeyavijayan Rajendran and Ahmad-Reza Sadeghi},
  year = {2026},
  month = jun,
  eprint = {2606.15809},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.15809}
}