Automating Safety Enhancement for LLM-based Agents with Synthetic Risk Scenarios

Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Yongtian Xu, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun · 2025

Read Automating Safety Enhancement for LLM-based Agents with Synthetic Risk Scenarios by Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Yongtian Xu, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17735.txt.

Description

AI agents built on large language models now handle real tasks, yet they can be steered into unsafe acts through tricky prompts and tools. Xueyang Zhou led a team that built AutoSafe, a system that makes agents safer without collecting dangerous real-world examples. It starts from a threat model that spells out how unsafe behavior grows out of user wording, context, and the agent's own steps.

From that model, the system invents risky scenes on its own, thinks through safe replies, and builds a large training set from them. Agents trained this way scored about forty-five percent higher on safety tests and did far better on real-world tasks too. The team released the project in public for others to build on.

Catalog record

CREATORXueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Yongtian Xu, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17735

Browse the full library · About the Tower of Babel