Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Junjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang · 2026
Read Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks by Junjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04459.txt.
Description
Chu and colleagues audit code quality and adoption in safety benchmarks. Benchmarks guide safety work, yet runnability and adoption drivers lacked systematic study. The team measures 31 benchmarks against 382 other papers with static analysis, human tests over 220 hours, and bibliometrics. Only 39 percent run unmodified, 16 percent give flawless install guides, and 6 percent address ethics despite harmful content. Adoption tracks prominence and runnability, not code quality, and some repos expose jailbreak outputs without warnings.
Catalog record
| CREATOR | Junjie Chu Xinyue Shen Ye Leng Michael Backes Yun Shen Yang Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79156 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04459