The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

Abstract

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 pl

Categories

@misc{vohra2026audit,
  title = {{The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits}},
  author = {Siddharth Vohra and Manikandan Ravikiran},
  year = {2026},
  month = sep,
  eprint = {2609.09048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.09048}
}