Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor

Abstract

Counterfactual audits are the standard tool for checking whether a clinical agent treats demographically distinct but clinically identical patients differently. They report a flip rate: how often an action changes when only the patient descriptor changes. We show that this quantity is uninterpretable on its own. Re-running an identical condition ten times over sixteen vignettes (same narrative, same descriptor string, nothing varied) moved a clinical agent's action in 8.7% of outcome-vignette ce

Categories

@misc{bellibatlu2026counterfactual,
  title = {{Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor}},
  author = {Rohith Reddy Bellibatlu and Manpreet Singh and Deepak Parashar and Rahul Joshi},
  year = {2026},
  month = sep,
  eprint = {2609.03221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.03221}
}