Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
Angel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected] · 2026
Read Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification by Angel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03336.txt.
Description
Angel Rodrigo Avelar Menendez, Yufeng Liu, and Xiaowu Dai warn that language model leaderboards look firmer than they are. Rankings come from pairwise human preferences and guide real choices about deployment and spending. Yet standard leaderboards print one fixed order, hiding estimation noise and the fact that a model's strength shifts with the kind of prompt. Acting on shaky gaps wastes resources and can mislead buyers.
Their 2026 framework ranks models per prompt type with honest uncertainty attached. A contextual preference model ties each model's appeal to prompt features, and simultaneous confidence intervals yield sets of plausible ranks rather than one brittle order. On large-scale human preference data, many celebrated rank gaps proved statistically indistinguishable. The method claims dominance only when data support it, otherwise returning a partial order.
Catalog record
| CREATOR | Angel Rodrigo Avelar Menendez [email protected], Yufeng Liu [email protected], Xiaowu Dai [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86621 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03336