Permutation-Consensus Listwise Judging for Robust Factuality Evaluation

Tianyi Huang, Nathan Huang, Justin Tang, Wenqian Chen, Elsa Fan · 2026

Read Permutation-Consensus Listwise Judging for Robust Factuality Evaluation by Tianyi Huang, Nathan Huang, Justin Tang, Wenqian Chen, Elsa Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20562.txt.

Description

Huang and co-authors test a fix for a flaw in LLM judges: the order of answers can sway the verdict. Their method, PCFJudge, runs one prompt on the same answer set in seven unlike orders. It pools the scores, ranks, and doubt marks into a single call. On a factuality test from RewardBench 2, the top pick rate rose from 86 to just over 91 percent for the GPT model. The Claude model rose from 86 to just under 90 percent. Order bias, they conclude, is a real and fixable error source.

Catalog record

CREATORTianyi Huang, Nathan Huang, Justin Tang, Wenqian Chen, Elsa Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28261
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20562

Browse the full library · About the Tower of Babel