Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning
Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng · 2024
Read Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning by Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18693.txt.
Description
Alex Beutel, Kai Xiao, Johannes Heidecke, and Lilian Weng tackle a hard trade off in a 2024 paper on automated red teaming of AI models. Red teaming hunts for rare ways a model can fail or be misused. Past methods found attacks that were either varied or strong, but seldom both at once.
Their approach splits the job in two. First, a large model drafts many distinct attack goals, with rule based checks that grade success for each goal. Next, a trained attacker learns to hit those goals while staying unlike its past tries. Tests on prompt injection and unsafe replies produced attacks that were both more varied and more often successful than past methods. Such attacks can then harden models in training and testing.
Catalog record
| CREATOR | Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18693