paperSeptember 2026Unreviewed

When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation

Pei-ke Zhu, Sidi Chang

Abstract

Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits, consumer surplus, and welfare---without instantiating the behavior named in the claim. We audit this risk in a multi-turn buyer--seller testbed for configurable hotel transactions. An initial implementation reported welfare gains from two marketplace guardrails of +87.4, +35.0, and +28.8 across a Qwen2.5 1.5B--14B ladder. It also gave guarded an

Cite

@misc{zhu2026whenb,
  title = {{When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation}},
  author = {Pei-ke Zhu and Sidi Chang},
  year = {2026},
  month = sep,
  eprint = {2609.01519},
  archivePrefix = {arXiv},
  url = {https://www.semanticscholar.org/paper/96c5c7670b65dee1adc74eab2b024fb568032514}
}