Large Language Model Selection with Limited Annotations

Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel · 2026

Read Large Language Model Selection with Limited Annotations by Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24981.txt.

Description

Yavuz Durmazkeser and colleagues introduce SELECT-LLM, the framework for active selection among large language models. SELECT-LLM finds a few queries whose annotations best identify the top model, scoring them by expected information gain from pairwise similarities of outputs. The rule uses only generated responses, so it works for open-weight and black-box models without weight access. It beats the strongest baseline across 23 datasets and 156 models. Annotation cost falls up to 81.8 percent for best-model selection.

Catalog record

CREATORYavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74822
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24981

Browse the full library · About the Tower of Babel