Rulers : Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation

Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong · 2026

Read Rulers : Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation by Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08654.txt.

Description

Yihan Hong and co-authors present Rulers, a framework for robust language model evaluation with locked rubrics. Their 2026 arXiv paper notes that LLM-as-a-Judge evaluation promises scalable rubric-based grading. Aligning frozen, black-box judges with human standards stays hard due to generation randomness. The authors reframe judge alignment as a criteria transfer problem. They isolate three failure modes: rubric instability from prompt sensitivity, unverifiable reasoning, and scale misalignment with human grading. Evidence-anchored scoring aims to fix all three for more trustworthy automatic grades.

Catalog record

CREATORYihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08654

Browse the full library · About the Tower of Babel