TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge
Cheng-Han Chiang, Hung-yi Lee, Michal Lukasik · 2025
Read TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge by Cheng-Han Chiang, Hung-yi Lee, Michal Lukasik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04381.txt.
Introduction
Cheng-Han Chiang, Hung-yi Lee, and Michal Lukasik improve the training of language models that grade other texts. In the popular model-as-judge setup, a model reads a piece of writing and assigns a score by a rubric. Standard training treats that score like a label to memorize. It ignores that scores are numbers with meaningful distances.
Their method, TRACT, trains in two stages. First the model learns to reason step by step toward a score. Then it trains further with a loss that respects the numeric scale, so near misses cost less than wild ones. Across four judging datasets and two base models, the approach beat earlier methods clearly. Taking any piece of the recipe away hurt the results, which the authors read as proof each part matters.
Catalog record
| CREATOR | Cheng-Han Chiang, Hung-yi Lee, Michal Lukasik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04381
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge Cheng-Han Chiang, Hung-yi Lee, Michal Lukasik Abstract The LLM-as-a-judge paradigm uses large language models (LLMs) for automated text evaluation, which assigns a score to the text based on some scoring rubrics. Existing methods for LLM-as-a-judge use cross-entropy (CE) loss for fine-tuning, which neglects the numeric nature of score prediction. Recent work addresses numerical prediction limitations of LLM fine-tuning through regression-aware fine-tuning, which, however, does not consider chain-of-thought (CoT) reasoning for score prediction. In this paper, we introduce TRACT ( Two-stage Regression-Aware fine-tuning with CoT ), a method combining CoT reasoning with regression-aware training. The training objective of TRACT combines the CE loss for learning the CoT reasoning and the regression-aware loss for the score prediction. TRACT consists of two stages: first, a seed LLM is fine-tuned to generate CoTs; next, we retrain the seed LLM using the CoTs generated by the LLM trained in stage 1. Experiments across four LLM-as-a-judge datasets and two LLMs show that TRACT significantly outperforms existing methods. Extensive ablation studies validate the importance of each component in TRACT. 1 1 1 All code and models can be found at https://github.com/d223302/TRACT . 1 Introduction Large Language Models (LLMs) have been applied to evaluate written based on the fine-grained evaluation rubrics specified in the input by outputting a score indicating the quality ( Chiang and Lee, 2023a ) . The fine-grained evaluation rubrics define the criteria for scoring answers, and each instance to be evaluated can have a set of customized fine-grained evaluation rubrics. This LLM-as-a-judge paradigm has become the standard way to evaluate LLMs ( Zheng et al., 2023 ; Dubois et al., 2024 ; Abdin et al., 2024 ; Lambert et al., 2025 ) , and has applications across diverse scenarios ( Yuan et al., 2024 ; Chiang et al., 2024 ) . To induce the fine-grained assessment ability of LLMs, LLM-as-a-judge is typically trained with cross-entropy (CE) loss to predict chain-of-thought (CoT) reasoning about the evaluation, followed by a score ( Kim et al., 2024a ; Kim et al., 2024b ; Li et al., 2024 ) . (a) CoT-RAFT fine-tuning objective (§ 3.2 ). (b) TRACT algorithm (§ 3.3 ). Figure 1: TRACT method overview. (a) Illustration of the CoT-RAFT fine-tuning objective (Eq. 4 ), used in both stages of TRACT. (b) Two fine-tuning stages of TRACT (also see Algorithm 1 ). Stage 1 : model p_{\rm s} is trained over the ground truth scores and the annotation CoTs (generated by the annotation model p_{\rm a} ). Stage 2 : CoT supervision is sampled from p_{\rm s} (frozen at this stage) and used to fine-tune the final model p_{\rm tract} . Although fine-tuning LLM-as-a-judge with CE loss is a well-established practice ( Kim et al., 2024a ; Kim et al., 2024b ) , it has intuitive limitations when applied to predicting numerical targets. For example, given a ground truth score ’ 1 ’, placing high probability on the token ’ 5 ’ is penalized the same as placing high probability on the token ’ 2 ’, despite the fact that they induce very different errors in terms of numerical metrics such as squared error. To mitigate this, Lukasik et al. (2025) proposed regression-aware fine-tuning (RAFT; see § 2.3 ), a method using squared error loss in fine-tuning LLMs. RAFT has shown promising results on regression tasks; however, it does not incorporate CoT reasoning Wei et al. (2022) ; Kojima et al. (2022) . Chiang and Lee (2023b) show that not using CoT can harm the performance of LLM-as-a-judge. This makes us ask: Can CoT reasoning be effectively integrated into regression-aware fine-tuning? In this paper, we propose T wo-stage R egression- A ware fine-tuning with C o T reasoning ( TRACT ), a method that combines the benefits of CoT reasoning with RAFT for improved numerical prediction abilities.. In stage 1 , TRACT fine-tunes the seed LLM over both the CoT supervision and the ground truth scores in the original training data. TRACT’s fine-tuning objective (which we refer to as CoT-RAFT) is the sum of CE loss over the CoT, and the RAFT loss over score prediction (see Figure 1(a) ). In stage 2 , TRACT fine-tunes the seed LLM from the CoTs generated by the model trained in stage 1 using the same CoT-RAFT objective. This two-stage pipeline is for reducing the distribution between the seed LLM and the CoT supervision used for fine-tuning in stage 2 (see Figure 1(b) ). We summarize TRACT in Algorithm 1 and contrast its formulation against baselines in Table 1 . We conduct experiments on five datasets and two LLMs and show that TRACT consistently outperforms baselines by a large margin, including Prometheus-2-7B ( Kim et al., 2024b ) , which is the best model of equal size from prior works. Our contributions are as follows: (i) We propose TRACT, a fine-tuning method that combines CoT reasoning with regression-aware loss to improve the numerical scoring ability of LLMs (§ 3 ). (ii) We show that TRACT yields significant performance improvements over existing baselines for LLM-as-a-judge. We validate the importance of each component in TRACT with extensive ablations (§ 5 , § 6 ). (iii) We show that TRACT, while trained on point-wise LLM-as-a-judge datasets, can work well on RewardBench ( Lambert et al., 2024 ) , a pairwise comparison dataset (§ 7 ). (iv) We release our models. Our best model surpasses the prior state-of-the-art model of comparable size by 0.05 in Pearson’s correlation when the inference-time compute is limited. Category Approach Predictor function {\hat{y}}(x) Fine-tuning loss Sec Baselines Standard fine-tuning and decoding w/o CoT \argmax_{y\in\mathcal{Y}}{{p}}(y\,|\,x) -\log{{p}}(y^{*}\,|\,x) § 2.2 RAIL zero-shot ( Lukasik et al., 2024 ) \sum_{y^{\prime}\in\mathcal{Y}}y^{\prime}\cdot{p}(y^{\prime}|x) N/A § 2.3 RAFT Lukasik et al. (2025) \sum_{y^{\prime}\in\mathcal{Y}}y^{\prime}\cdot{p}(y^{\prime}|x) \left(\hat{y}(x)-y^{*}\right)^{2} § 2.3 Standard fine-tuning and decoding w/ CoT \argmax_{y\in\mathcal{Y}}p(\texttt{str}(y)\,|\,[x,\hat{s}]) ; \hat{s}\sim p(\cdot\,|\,x) -\log{p}([s^{*},y^{*}]\,|\,x) § 3.1 Ours TRACT \sum_{y\in\mathcal{Y}}p(\texttt{str}(y)\,|\,[x,\hat{s}])\cdot y ; \hat{s}\sim p(\cdot\,|\,x) Algorithm 1 § 3 Table 1: Approaches to autoregressive regression with decoder-based LLMs. Here, p(\cdot\,|\,x) denotes a distribution over possible outputs given an input string x , and \hat{y}(x)\in\mathbb{R} a predictor function. \mathcal{Y} denotes the targets space (unless otherwise stated, \{1,2,3,4,5\} ). 2 Background We first introduce the notation and next review previous works on regression-aware numerical output prediction with autoregressive models. 2.1 Notation Let \mathcal{V} denote a finite vocabulary of tokens , \mathcal{X}\subseteq\mathcal{V}^{*} be a set of inputs comprising strings of tokens, \mathcal{S}\subseteq\mathcal{V}^{*} the set of CoT reasonings, and \mathcal{Y}\subset\mathbb{R} be a set of numerical targets . We assume that each y\in\mathcal{Y} has a unique string representation {\tt str}(y)\in\mathcal{V}^{*} . Let \mathbb{P} denote a ground-truth distribution over \mathcal{X}\times\mathcal{S}\times\mathcal{Y} , with the decomposition \mathbb{P}(x,s,y)=\mathbb{P}(x)\cdot\mathbb{P}(s\,|\,x)\cdot\mathbb{P}(y\,|\,[x,s]) , where [x,s] denotes a string concatenation of x and s . Denote the training data by D_{\rm train}\in(\mathcal{X}\times\mathcal{Y})^{N} , and each element in {D}_{\rm train} is composed of (x,y^{*}) . Denote the training data augmented with CoT by D_{\rm train}^{\rm CoT}\in(\mathcal{X}\times\mathcal{S}\times\mathcal{Y})^{N} , and each element in {D}_{\rm train}^{\rm CoT} is composed of (x,s^{*},y^{*}) , where s^{*} can come from the human or LLM annotation. In this paper, we focus on LLM-as-a-judge application; in this case, the input {x}\in\mathcal{X} consists of the instructions for LLM as a judge, the scoring rubrics and a sample to be evaluated.