Best Arm Identification with LLM Judges and Limited Human Audits

Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology · 2026

Read Best Arm Identification with LLM Judges and Limited Human Audits by Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21471.txt.

Description

Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, and David Simchi-Levi study how to find the best option when judges are cheap but biased. A language model can grade many samples at low cost. True labels from human audits are accurate but expensive, so they can only be bought for a few samples.

Unlike classic setups, the cheap judge is biased in ways that depend on the option and the context. Ground truth also arrives only where the team chooses to look. The authors design a method with firm guarantees for this setting. It fits modern evaluation, where AI judges do the bulk grading.

Catalog record

CREATORRuicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi, [2ex] Institute for Data, Systems, Society, Massachusetts Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21471

Browse the full library · About the Tower of Babel