BenchScope: How Many Independent Signals Does Your Benchmark Provide?

Tommy Sha *, Stella Zhao * · 2026

Read BenchScope: How Many Independent Signals Does Your Benchmark Provide? by Tommy Sha *, Stella Zhao * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29357.txt.

Description

Sha and Zhao ask how many independent signals a benchmark really gives. Score suites often report many numbers that repeat the same information. They define Effective Dimensionality from the spectrum of benchmark scores. It acts as a fast upper-bound check on measurement breadth. They applied it to 22 benchmarks and over 8,400 model runs. The six-score Open LLM Leaderboard behaved like about two axes. BBH and MMLU-Pro carried near-identical signal. The authors release an atlas and a four-step workflow.

Catalog record

CREATORTommy Sha *, Stella Zhao *
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80808
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29357

Browse the full library · About the Tower of Babel