RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models

Quy-Anh Dang, Chris Ngo, Truong-Son Hy · 2026

Read RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models by Quy-Anh Dang, Chris Ngo, Truong-Son Hy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03699.txt.

Description

Quy-Anh Dang, Chris Ngo, and Truong-Son Hy introduce RedBench, a universal dataset for red teaming language models, meaning systematic attempts to provoke unsafe outputs. Safety testing has been hampered by inconsistent risk categories, narrow coverage, and stale evaluations across separate benchmarks. RedBench merges thirty-seven existing datasets from major conferences and repositories into one resource.

The result holds nearly thirty thousand samples of attack and refusal prompts. A standard taxonomy organizes them into twenty-two risk categories across nineteen domains, enabling consistent comparisons between models. The authors analyze the source datasets, set baselines for modern models, and release the data and evaluation code openly. Shared, current red-teaming material helps the whole field measure vulnerabilities the same way instead of each lab inventing its own yardstick.

Catalog record

CREATORQuy-Anh Dang, Chris Ngo, Truong-Son Hy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03699

Browse the full library · About the Tower of Babel