RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun · 2026
Read RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty by Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12424.txt.
Description
Benchmarks rank language models, but most treat every question as equally hard, blurring real capability gaps. Ziqian Zhang led this work with ten co-authors, presenting RankLLM, a frame that scores both question difficulty and model skill together. Difficulty becomes the main lens for telling models apart, rather than a raw count of correct answers.
Scores flow both ways: a model earns skill credit for answering a question, while a question gains difficulty credit when it stumps a model. Tested on 30 models and 35,550 questions across domains, RankLLM agrees with human judgements 90 percent of the time, beating strong baselines such as IRT. It also shows stability, fast convergence, and high efficiency, making difficulty-aware ranking practical at large scale.
Catalog record
| CREATOR | Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12424