Training Language Model to Critique for Better Refinement
Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang · 2025
Read Training Language Model to Critique for Better Refinement by Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22157.txt.
Introduction
Tianshu Yu and co-authors present, in a 2025 machine learning paper, a training scheme that teaches models to write critiques which actually help another model improve its answers. Good feedback is a known strength of language models, yet nobody had pinned down which critiques drive real fixes. Their loop rewards the critic by the quality of the fix its words produce.
Each critique is scored by how much the revised answer improves, and that score trains the critic, with no need for human rankings of the critiques themselves. Across dialogue, summaries, questions, math, and code, the method beat standard training and open rivals on both critique quality and fix quality.
Catalog record
| CREATOR | Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Language Model to Critique for Better Refinement Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang Abstract Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce R efinement-oriented C ritique O ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method’s effectiveness in enhancing LLM critique-refinement loops. 1 1 1 We release the code and data at https://github.com/publicstaticvo/critique. 1 Introduction The critique ability, defined as the capacity to identify and refine flaws in responses, is crucial for improving the reliability of automatic evaluation and enabling the self-improvement of large language models (LLMs) Saunders et al. 2022 ; Scheurer et al. 2023 . Enhancing this ability is challenging, as it requires LLMs to not only detect errors but also engage in a deeper analysis of user queries and responses, extending beyond superficial criticism Zheng et al. 2023b ; Kim et al. 2024 . Recent studies have sought to address this challenge of LLMs via human-curated critique datasets and alignment strategies like supervised fine-tuning (SFT) Cui et al. 2024 ; Ke et al. 2024 and reinforcement learning Akyürek et al. 2023 ; McAleese et al. 2024 , yielding promising results in identifying and rectifying shortcomings across various tasks. Figure 1: A comparison between previous methods, which train critic models via direct critique preference judgment and are not helpful for refinement, and our method, which train critic models by converting preferences of refined responses into critique reward values and is helpful for refinement. However, current methods primarily train models to generate critiques for evaluation purposes, rather than linking critique to refinement Li et al. 2024b . This disjointed approach fails to connect error identification with output improvement, limiting the practical value of critiques. While the critique-refinement paradigm is explored in LLM alignment Madaan et al. 2023 ; Wadhwa et al. 2024 and reasoning Chen et al. 2024 ; Wu et al. 2024 , existing works lack systematic investigation on how critique enhancements can lead to meaningful refinements in responses. Without this link, assessing critique quality remains challenging Sun et al. 2024 . Addressing this gap is essential for advancing both LLM evaluation and self-improvement capabilities. To address these challenges, we introduce Refinement-oriented Critique Optimization (RCO), a novel training paradigm for critic models that uses critique utility (CU) as a reward signal, which is calculated by comparing the refined responses to the initial ones. This approach encourages the generated critiques to drive substantial improvements in the output, creating a more effective critique-refinement loop. Specifically, RCO feeds critiques and initial responses into the actor model and prompts it to generate multiple refined responses. CU is then quantified as the proportion of refinements preferred over the initial response, which serves as the reward signal for training the critic model. By focusing on the refinement outcomes, RCO eliminates the need for direct critique preference assessment and ensures that critiques leading to better refinements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, using baseline models, models trained with critique preferences, and advanced open-source models. Experimental results demonstrate that RCO significantly enhances critic model performance, outperforming existing methods across multiple benchmarks. Our contributions are threefold: (1) We propose RCO, a method that prioritizes critiques that facilitate effective refinement of actor model responses, addressing limitations of previous approaches. (2) We introduce a novel supervision scheme based on refined response preferences, eliminating the need for directly assessing the quality of critiques while rewarding critiques that lead to meaningful improvements. (3) We rigorously evaluate RCO across diverse tasks, showing substantial improvements in critique quality and refinement capabilities, and provide an in-depth analysis of the method’s impact. Figure 2: The illustration of our method RCO, describing our data collection and training process. We first generate critiques with the critic model, then sample multiple refined responses with the actor model. Subsequently, we calculate the critique utility of each critique with preference judgment on refined and initial responses, which serves as reward signal of critic model training. 2 Related Work Critique Ability of LLMs The rapid advancement of large language models (LLMs) has highlighted the need to enhance their critique capabilities. LLMs like GPT-4 Achiam et al. 2023 have proven effective as evaluators Zheng et al. 2023a ; Li et al. 2024a ; Cao et al. 2024 , but their API-based access limits widespread application. To address cost and stability issues, researchers have fine-tuned open-source models using critique data from these models. Despite these efforts, challenges in critique task complexity remain. Recent work includes Murugadoss et al. 2024 , which uses prompt engineering to create critique metrics, and Verga et al. 2024 , which proposes using multiple LLMs to mitigate bias. Ke et al. 2024 generates a golden critique dataset from model and reference response pairs, while Lan et al. 2024a employs a multi-agent framework for preference-based critique data collection. In contrast, our method improves critique quality by automatically obtaining preferences of refined responses to calculate critique utility as reward signals to train critic models. Preference-Based Reinforcement Learning Reinforcement Learning from Human Feedback (RLHF) Ziegler et al. 2019 is commonly used to guide LLMs towards human-preferred responses. Scheurer et al. 2023 uses RLHF to train reward models based on human-annotated pairwise comparisons. Recent methods, such as CriticGPT McAleese et al. 2024 , apply RLHF to enhance critique abilities by training critic models on human-identified errors in code generation. Similarly, Wang et al. 2024b collects preference critique pairs by comparing LLM-generated responses with human-annotated scores. These approaches are limited by the high cost and uncertain quality of human annotations Sun et al. 2024 . In contrast, our approach reduces reliance on human annotations, providing a clear standard for good critiques that effectively guide actor model refinements. 3 Methodology The overview of RCO is illustrated in Figure 2 . Our approach begins with a dataset \mathcal{D} , where each sample \mathbf{X}=(x,y_{0}) consists of a prompt x and its corresponding initial response y_{0} , with y_{0} being generated by an actor model \pi(y_{0}|x) . Following this, a base critic model p(c|y_{0},x) is employed to generate N distinct critiques, denoted as c_{1},c_{2},\dots,c_{N} , for the initial