ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen · 2024

Read ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities by Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06745.txt.

Description

Adhiraj Ghosh and co-authors propose a new way to test foundation models on open-ended skills. Fixed test sets miss the broad skills such models now claim. Their frame, ONEBench, pools samples from many test sets into one growing bank. Users can then draw custom tests for the skill they care about.

Pooling fights overfitting to one set and widens what can be checked. Two snags arise: scores mix unlike measures, and models may sit distinct subsets. Their pooling algorithm recovers true scores in the limit and settles fast. It held rankings even with ninety-five percent of scores missing. Cuts in cost reached twenty times with no rank change. Versions serve text and vision-language models.

Catalog record

CREATORAdhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao Samuel Albanie, Matthias Bethge, Tübingen AI Centre, University of Tübingen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06745

Browse the full library · About the Tower of Babel