Large Language Model Selection with Limited Annotations
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel · 2026
Read Large Language Model Selection with Limited Annotations by Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24981.txt.
Description
Yavuz Durmazkeser and colleagues introduce SELECT-LLM, the framework for active selection among large language models. SELECT-LLM finds a few queries whose annotations best identify the top model, scoring them by expected information gain from pairwise similarities of outputs. The rule uses only generated responses, so it works for open-weight and black-box models without weight access. It beats the strongest baseline across 23 datasets and 156 models. Annotation cost falls up to 81.8 percent for best-model selection.
Catalog record
| CREATOR | Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74822 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24981