paperAugust 2026Unreviewed

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

Han Wang, Alex P. Whitworth, Pak-Ming Cheung, Zhenjie Zhang, Krishna Kamath, Xi Chen, Roberto Konow, Kurchi Subhra Hazra

View resource

Abstract

Relevance evaluation plays a crucial role in personalized search systems, serving as a guardrail alongside user engagement metrics to ensure that search results align with user queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present a VLM-based automated relevance evaluation pipeline deployed within Pinterest Search for online A/B experiments. We rigorously validate the a

Cite

@misc{wang2026advancing,
  title = {{Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search}},
  author = {Han Wang and Alex P. Whitworth and Pak-Ming Cheung and Zhenjie Zhang and Krishna Kamath and Xi Chen and Roberto Konow and Kurchi Subhra Hazra},
  year = {2026},
  month = aug,
  eprint = {2608.02446},
  archivePrefix = {arXiv},
  doi = {10.1145/3773078.3831891},
  url = {https://www.semanticscholar.org/paper/07bc043030832c7a90b135a17b052ac818a21920}
}