ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities
Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen · 2024
Read ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities by Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06745.txt.
Description
Adhiraj Ghosh and co-authors propose a new way to test foundation models on open-ended skills. Fixed test sets miss the broad skills such models now claim. Their frame, ONEBench, pools samples from many test sets into one growing bank. Users can then draw custom tests for the skill they care about.
Pooling fights overfitting to one set and widens what can be checked. Two snags arise: scores mix unlike measures, and models may sit distinct subsets. Their pooling algorithm recovers true scores in the limit and settles fast. It held rankings even with ninety-five percent of scores missing. Cuts in cost reached twenty times with no rank change. Versions serve text and vision-language models.
Catalog record
| CREATOR | Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06745