YourBench: Easy Custom Evaluation Sets for Everyone
Sumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür · 2025
Read YourBench: Easy Custom Evaluation Sets for Everyone by Sumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01833.txt.
Description
Sumuk Shashidhar and co-authors present a framework that builds fresh test sets for language models from user documents. Their 2025 system tackles a known crunch: fixed benchmarks saturate and leak into training data, while human grading is slow and dear. Domain-fit tests are hard to get on time.
The tool drafts tailored questions with no hand labeling. It copied seven test subsets from minimal source text for a small total cost, and model rankings matched the source tests exactly. A new set of post-cutoff documents guards against models answering from memory. Checks span many top models and human review. All code, data, and traces are shared in public.
Catalog record
| CREATOR | Sumuk Shashidhar, Clementine Fourier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, Dilek Hakkani-Tür |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01833