Automatic LLM Red Teaming
Roman Belaire, Arunesh Sinha, Pradeep Varakantham · 2025
Read Automatic LLM Red Teaming by Roman Belaire, Arunesh Sinha, Pradeep Varakantham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04451.txt.
Description
Roman Belaire, Arunesh Sinha, and Pradeep Varakantham present a stronger automated way to attack chatbots in order to find their weak points before real attackers do, in a 2025 paper. The head title stresses that such dialogue attacks come in levels. Older test methods lean on fixed templates and single messages, which miss how real harmful chats build over turns.
Their system treats the test as a long game of choices, learned at two levels: broad strategy and word-by-word moves. A reward tied to the harm in each reply guides the learning, even when feedback is sparse. The trained tester found bypasses that template methods missed and set a new best result. The authors frame long, planned test chats as the right way to judge chatbot safety.
Catalog record
| CREATOR | Roman Belaire, Arunesh Sinha, Pradeep Varakantham |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04451