Best Arm Identification with LLM Judges and Limited Human Audits
Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology · 2026
Read Best Arm Identification with LLM Judges and Limited Human Audits by Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21471.txt.
Description
Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, and David Simchi-Levi study how to find the best option when judges are cheap but biased. A language model can grade many samples at low cost. True labels from human audits are accurate but expensive, so they can only be bought for a few samples.
Unlike classic setups, the cheap judge is biased in ways that depend on the option and the context. Ground truth also arrives only where the team chooses to look. The authors design a method with firm guarantees for this setting. It fits modern evaluation, where AI judges do the bulk grading.
Catalog record
| CREATOR | Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21471