Skip to content
Search
paperJune 2026Unreviewed

HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice

Sarah Meiklejohn, Sunny Consolvo, Patrick Gage Kelley, Tara Matthews, Sai Teja Peddinti, Renee Shelby, Lenin Simicich, Kurt Thomas

View resourcePDF

Abstract

This paper introduces HelpBench, a benchmark for assessing whether LLMs are capable of providing accurate help in response to questions about digital privacy, safety, and security. We curated 450 questions representing authentic user situations and developed rubrics for each question to evaluate the factual accuracy and tone of a response. Example questions touch on how to regain access to lost or suspended accounts, how to balance the trade-offs of hardware security keys versus other forms of t

Categories

Benchmarks & Evaluation

Cite

@misc{meiklejohn2026helpbench,
  title = {{HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice}},
  author = {Sarah Meiklejohn and Sunny Consolvo and Patrick Gage Kelley and Tara Matthews and Sai Teja Peddinti and Renee Shelby and Lenin Simicich and Kurt Thomas},
  year = {2026},
  month = jun,
  eprint = {2606.24819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.24819}
}