Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng · 2024

Read Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning by Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18693.txt.

Description

Alex Beutel, Kai Xiao, Johannes Heidecke, and Lilian Weng tackle a hard trade off in a 2024 paper on automated red teaming of AI models. Red teaming hunts for rare ways a model can fail or be misused. Past methods found attacks that were either varied or strong, but seldom both at once.

Their approach splits the job in two. First, a large model drafts many distinct attack goals, with rule based checks that grade success for each goal. Next, a trained attacker learns to hit those goals while staying unlike its past tries. Tests on prompt injection and unsafe replies produced attacks that were both more varied and more often successful than past methods. Such attacks can then harden models in training and testing.

Catalog record

CREATORAlex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18693

Browse the full library · About the Tower of Babel