Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks

Junjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang · 2026

Read Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks by Junjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04459.txt.

Description

Chu and colleagues audit code quality and adoption in safety benchmarks. Benchmarks guide safety work, yet runnability and adoption drivers lacked systematic study. The team measures 31 benchmarks against 382 other papers with static analysis, human tests over 220 hours, and bibliometrics. Only 39 percent run unmodified, 16 percent give flawless install guides, and 6 percent address ethics despite harmful content. Adoption tracks prominence and runnability, not code quality, and some repos expose jailbreak outputs without warnings.

Catalog record

CREATORJunjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79156
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04459

Browse the full library · About the Tower of Babel