paperNovember 2023Unreviewed

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Bhaktipriya Radharapu, Kevin Robinson, L. Aroyo, Preethi Lahoti

Conference on Empirical Methods in Natural Language Processing

Abstract

Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AI-assisted Red-Teaming (AART) - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce human effort signific

Framework mappings

NIST AI Risk Management Framework

MEASUREMeasure

Suggested from the entry's categories.

Cite

@inproceedings{radharapu2023aart,
  title = {{AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications}},
  author = {Bhaktipriya Radharapu and Kevin Robinson and L. Aroyo and Preethi Lahoti},
  year = {2023},
  month = nov,
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  eprint = {2311.08592},
  archivePrefix = {arXiv},
  doi = {10.48550/arXiv.2311.08592},
  url = {https://www.semanticscholar.org/paper/57d0e672040800e8d882ff0022647c087095e35f}
}