Through the Judge’s Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generat

Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei · 2025

Read Through the Judge’s Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generat by Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25860.txt.

Introduction

Xingjian Zhang leads six co-authors in work on making language models more reliable when they rate the work of others. For subjective tasks, a plain label hides the subtle reasoning behind a human judgment. The reasoning, called a thinking trace, would help train better raters, but traces are hard to collect and clean at scale.

Zhang and the team infer traces from label only data with humans and models working together, using a rejection sampling step to keep good reconstructions. They use the traces to fine tune open raters and to write clearer guidelines for closed raters. Agreement between models and humans rose across data sets, and model to model agreement rose as well.

Catalog record

CREATORXingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25860

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Through the Judge’s Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei Abstract . Large language models (LLMs) are increasingly used as raters for evaluation tasks. However, their reliability is often limited for subjective tasks, when human judgments involve subtle reasoning beyond annotation labels. Thinking traces , the reasoning behind a judgment, are highly informative but challenging to collect and curate. We present a human-LLM collaborative framework to infer thinking traces from label-only annotations. The proposed framework uses a simple and effective rejection sampling method to reconstruct these traces at scale. These inferred thinking traces are applied to two complementary tasks: (1) fine-tuning open LLM raters; and (2) synthesizing clearer annotation guidelines for proprietary LLM raters. Across multiple datasets, our methods lead to significantly improved LLM-human agreement. Additionally, the refined annotation guidelines increase agreement among different LLM models. These results suggest that LLMs can serve as practical proxies for otherwise unrevealed human thinking traces, enabling label-only corpora to be extended into thinking–trace–augmented resources that enhance the reliability of LLM raters. 1 1 1 For reproducing results, our codebase can be accessed at: https://github.com/xingjian-zhang/thru_judge_eye . 1. Introduction Large language models (LLMs) are increasingly used as automated evaluators for open-ended text generation tasks, primarily due to their remarkable efficiency and scalability ( Gu et al. 2024a ; Zheng et al. 2023 ) . This approach has been applied to evaluate diverse applications such as code generation ( Zhou et al. 2025 ) , text summarization ( Bedemariam et al. 2025 ) , and dialogue ( Gu et al. 2024a ) . However, the reliability of LLM evaluators often decreases in subjective tasks that require nuanced human judgment ( Ismayilzada et al. 2024 ; Gómez-Rodríguez and Williams 2023 ) , especially when they did not receive specialized training ( Krumdick et al. 2025 ) . To understand what is missing in calibrating these models, we can draw an analogy from the process of training human annotators for qualitative coding tasks. To ensure alignment on a subjective task, a novice annotator typically benefits from three key components that an expert can provide: (1) an annotation codebook with detailed instructions and rubrics; (2) a set of examples illustrating representative inputs and their corresponding labels; and critically, (3) explanations detailing the reasoning for assigning a specific label to each example. Among the three components, explanations are particularly important in creating a shared understanding of the task among annotators ( Artstein and Poesio 2008 ; O’Connor and Joffe 2020 ) . We hypothesize that these explanations of the expert’s reasoning, which we term the thinking traces , are equally necessary to calibrate LLM evaluators and improve their alignment with human judgment. Despite their value, thinking traces are largely absent from existing annotation datasets. The primary reason is the substantial effort required: articulating and recording a detailed reasoning process is significantly more time-consuming and costly than providing a single label ( DeYoung et al. 2019 ) . Most annotation interfaces are not designed to capture this information, and incentivizing raters to produce high-quality traces is challenging ( Carton et al. 2020 ) . As a consequence, many datasets for subjective tasks are limited to sparse and sometimes unreliable labels, lacking the rich context that thinking traces would provide. Reasoning language models (RLMs) offer a promising opportunity to address this issue ( Comanici et al. 2025 ; Guo et al. 2025 ; Jaech et al. 2024 ) . These models are designed to generate step-by-step reasoning, often termed “intermediate reasoning tokens”, before producing a final answer ( Guo et al. 2025 ) . This model-generated reasoning is intended to align with the underlying thinking traces of human experts, also known as “human priors” in Guo et al. 2025 . If this alignment is faithful, these models could serve as a proxy to reconstruct thinking traces at scale, mitigating the costly human annotation process. However, whether AI-generated reasoning truly aligns with human reasoning remains an open question, especially for subjective tasks where reasoning can be highly nuanced. While directly verifying this alignment is challenging, given the lack of ground-truth human thinking traces, a more pragmatic research question in our context is: Can these model-inferred thinking traces, even if imperfect, serve as a useful proxy to improve the reliability of automated LLM raters? Please read the prompt, the human story and the subject story (both stories might be the same). The story you will have to rate is the subject story. Note: some stories have been abruptly cut in the middle of a sentence. Please rate them as if they ended just before the unfinished sentence. Note: if the story is not relevant with respect to the prompt, it only affects the Relevance criterion! Do not rate 1 everywhere. Then, please rate the subject story on a scale from 1 (worst) to 5 (best) on Complexity: 1: The story is very simple with no elaborate elements. 2: The story is somewhat simple with few elaborate elements. 3: The story has some complexity but lacks depth in certain areas. 4: The story is mostly complex with minor areas lacking depth. 5: The story is highly complex and elaborate throughout. Figure 1. Original Annotation Codebook (Evaluating Complexity of Short Stories) ( Chhun et al. 2022 ) . Only very basic instructions and scoring rubrics are provided. To tackle this question, we present a human-LLM collaborative framework to infer thinking traces from label-only human annotations. We employ a simple, yet effective, rejection sampling method to reconstruct these traces at scale. We demonstrate the utility of inferred thinking traces through two complementary applications. First, we use them as training data to fine-tune open-weight LLM raters so that they are specialized for the evaluation task. Second, for models where fine-tuning is not an option, we introduce a novel method for codebook refinement. Inspired by iterative refinement practices in qualitative coding ( O’Connor and Joffe 2020 ) , this approach addresses the common problem of ambiguous instructions in many annotation codebooks ( Klie et al. 2023 ) (Figure 1 ). Our automatic pipeline leverages the inferred traces to synthesize a new, more explicit codebook that better steers the model’s judgments. Experiments show that both methods significantly improve the alignment of LLM raters with human judgments. Furthermore, the synthesized codebook also improves the agreement between different LLM providers, demonstrating its utility in promoting cross-model consistency. These findings demonstrate that model-inferred thinking traces can effectively serve as a practical proxy for human reasoning, thus improving the reliability and consistency of automated LLM raters on subjective evaluation tasks. Okay, let’s tackle this rating. The user wants me to evaluate the Subject Story based on the Engagement criterion. The prompt mentions that

Read the complete text