Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification

Angel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected] · 2026

Read Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification by Angel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03336.txt.

Description

Angel Rodrigo Avelar Menendez, Yufeng Liu, and Xiaowu Dai warn that language model leaderboards look firmer than they are. Rankings come from pairwise human preferences and guide real choices about deployment and spending. Yet standard leaderboards print one fixed order, hiding estimation noise and the fact that a model's strength shifts with the kind of prompt. Acting on shaky gaps wastes resources and can mislead buyers.

Their 2026 framework ranks models per prompt type with honest uncertainty attached. A contextual preference model ties each model's appeal to prompt features, and simultaneous confidence intervals yield sets of plausible ranks rather than one brittle order. On large-scale human preference data, many celebrated rank gaps proved statistically indistinguishable. The method claims dominance only when data support it, otherwise returning a partial order.

Catalog record

CREATORAngel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86621
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03336

Browse the full library · About the Tower of Babel