YourBench: Easy Custom Evaluation Sets for Everyone

Sumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür · 2025

Read YourBench: Easy Custom Evaluation Sets for Everyone by Sumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01833.txt.

Description

Sumuk Shashidhar and co-authors present a framework that builds fresh test sets for language models from user documents. Their 2025 system tackles a known crunch: fixed benchmarks saturate and leak into training data, while human grading is slow and dear. Domain-fit tests are hard to get on time.

The tool drafts tailored questions with no hand labeling. It copied seven test subsets from minimal source text for a small total cost, and model rankings matched the source tests exactly. A new set of post-cutoff documents guards against models answering from memory. Checks span many top models and human review. All code, data, and traces are shared in public.

Catalog record

CREATORSumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE111 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01833

Browse the full library · About the Tower of Babel