A Finite-Calibration Regime Map for LLM Judge Panels
Bin Zhu, Yanghui Rao · 2026
Read A Finite-Calibration Regime Map for LLM Judge Panels by Bin Zhu, Yanghui Rao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01034.txt.
Description
Zhu, Xie, and Rao map when a panel of AI judges can be trusted with few human labels in this 2026 paper. A judge panel grades outputs, but humans must supply labels to calibrate it. Labels get spent on fitting a calibrator, building judge paths, and picking which setup to ship. The team asks when a small model of the panel beats a full joint table.
Their method, FCPS, picks the judge path, panel size, and aggregator as one validated choice. Support checks flag patterns the labels never saw. Across four datasets and a seven judge pool, simple aggregation beats joint tables in 16 of 20 cells. Richer shrunk tables close some gaps but keep the core bottleneck. In grown data with a six way interaction, the map flips. The table grows to hold the interaction, and its error score falls from 0.224 down to 0.061 once the unseen mass vanishes.
Catalog record
| CREATOR | Bin Zhu, Yanghui Rao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01034