Skip to content
Search
paperSeptember 2026Unreviewed

Commit-first LLM judging inherits the judge's own errors

Idil Gozel

View resourcePDF

Abstract

LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. N

Categories

Benchmarks & Evaluation

Cite

@misc{gozel2026commitfirst,
  title = {{Commit-first LLM judging inherits the judge's own errors}},
  author = {Idil Gozel},
  year = {2026},
  month = sep,
  eprint = {2609.00088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.00088}
}