Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

Yubo Wang, Xiang Yue, Wenhu Chen · 2025

Read Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate by Yubo Wang, Xiang Yue, Wenhu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17703.txt.

Introduction

Yubo Wang, Xiang Yue, and Wenhu Chen train language models to critique answers instead of copying them. Their twenty twenty-five paper proposes critique fine-tuning. Standard training makes models imitate correct responses to instructions. The authors argue that judging flawed answers builds deeper understanding.

Training data pairs noisy answers with teacher-written critiques. Several maths and instruction datasets were rebuilt in that form. Models trained this way beat imitation-trained models on reasoning tests. Gains appeared across six maths benchmarks and several base models. One model matched strong rivals after just one hour of training. The work suggests error analysis may teach more than perfect examples alone.

Catalog record

CREATORYubo Wang, Xiang Yue, Wenhu Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17703

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate Yubo Wang, Xiang Yue, Wenhu Chen Abstract Supervised Fine-Tuning (SFT) is commonly used to train language models to imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning ( CFT ), a method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding—traits often overlooked by standard SFT. To validate the effectiveness of CFT , we construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath), where GPT-4o serves as the teacher to generate critiques in the form of ([query; noisy response], critique). Experiments on these datasets demonstrate that CFT consistently outperforms SFT by 4–10% across six mathematical reasoning benchmarks, and is effective across different base models including Qwen2.5, Qwen2.5-Math, and DeepSeek-Math. Notably, our model Qwen2.5-Math- CFT only requires 1 hour of training on 8xH100 over the 50K examples, yet matches or outperforms strong competitors like Qwen2.5-Math-Instruct on most benchmarks, which use over 2M samples. Moreover, it matches the performance of SimpleRL, which is a DeepSeek-r1 replication trained with 140x more compute. Experiments on IF_Eval and MT-Bench further demonstrate that CFT can significantly enhance the model’s general generation and instruction-following capabilities, outperforming the Qwen2.5-Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models. https://tiger-ai-lab.github.io/CritiqueFineTuning/ Figure 1 : Comparison between CFT and SFT on 50K samples from WebInstruct ( Yue et al., 2024b ) . SFT-verified means SFT training on the responses validated by GPT-4o, SFT-GPT4o means SFT training on the responses from GPT-4o. CFT is our approach, which trains on the critique provided by GPT-4o. 1 Introduction Recently, large language models (LLMs) ( Achiam et al., 2023 ; Team et al., 2023 ; Dubey et al., 2024 ) have shown unprecedented performance on real-world problems. One core techniques is supervised fine-tuning (SFT), which trains LLMs to follow natural language instructions ( Wei et al., 2022 ; Ouyang et al., 2022 ; Sanh et al., 2022 ) . In the process of SFT, LLMs are forced to imitate the annotated responses . Numerous efforts have been made to build high-quality SFT datasets using approaches like Self-Instruct ( Wang et al., 2023b ) and Evol-Instruct ( Xu et al., 2024 ) to enhance LLMs’ general instruction-following capabilities. More recently, works such as MAmmoTH ( Yue et al., 2024a ; Yue et al., 2024b ) , MetaMath ( Yu et al., 2024 ) , and WizardCoder ( Luo et al., 2024 ) have employed SFT to improve the targeted capabilities of LLMs in areas like mathematical reasoning, coding, and more. While these approaches have shown significant gains on weaker base models such as Mistral ( Jiang et al., 2023 ) or LLaMA3 ( Dubey et al., 2024 ) , diminishing returns become evident as SFT dataset size and quality scale up. This limitation is particularly pronounced for already-powerful base models (non-SFTed), such as Qwen2.5-base ( Yang et al., 2024a ) , Qwen2.5-Math-base ( Yang et al., 2024b ) , or DeepSeek-Coder-V2-base ( Guo et al., 2024 ) , which have undergone extensive domain-adaptive pretraining on reasoning-focused corpora comprising hundreds of billions of tokens. Our experiments in section 3 reveal that applying SFT to these models can even degrade performance without stringent quality control. In this paper, we propose a new learning framework called Critique Fine-Tuning ( CFT ) as an alternative to traditional SFT. Inspired by human learning—where critical thinking and constructive feedback are vital for improvement—we shift the focus from imitation to critique-based learning. When humans learn, they do not merely replicate provided answers but analyze, critique, and refine them. Similarly, in CFT , the model learns to provide critiques for noisy responses, identify flaws, suggest improvements, and verify correctness. Formally, CFT involves training the model to critique a given query-response pair, maximizing the likelihood P(c|[x;y]) , where c is the annotated critique for a query-response pair [x;y] . A detailed visualization of CFT is presented in Figure 1 . To validate CFT ’s effectiveness, we designed a series of experiments using multiple critique datasets, including WebInstruct ( Yue et al., 2024b ) , MetaMathQA ( Yu et al., 2024 ) , and NuminaMath ( Li et al., 2024b ) , with critiques synthesized by advanced models such as GPT-4o ( Achiam et al., 2023 ) . We applied CFT to strong 7B base language models (non-instruction-tuned), including DeepSeekMath-base ( Shao et al., 2024 ) , Qwen2.5 ( Yang et al., 2024a ) , and Qwen2.5-Math ( Yang et al., 2024b ) . These models were compared against strong SFT-trained variants, such as WebInstruct-verified (SFT on GPT-4o-verified responses) and WebInstruct-GPT4o (SFT directly on GPT-4o-generated responses). When evaluated on six math benchmarks, including MATH and AIME24, CFT -trained models consistently outperform the best SFT-trained models by an average of 4–10 absolute points. We further expanded the evaluation to broader STEM benchmarks, including GPQA ( Rein et al., 2023 ) , TheoremQA ( Chen et al., 2023 ) , and MMLU-Pro ( Wang et al., 2024b ) . Our results show that the best CFT -trained model, Qwen2.5-Math- CFT , trained on 50K examples, outperformed strong competitors like AceMath ( Liu et al., 2024 ) and Qwen2.5-Math-Instruct ( Yang et al., 2024b ) , which were trained on over 2M examples. We also compare Qwen2.5-Math- CFT with SimpleRL ( Zeng et al., 2025 ) , an open replication of DeepSeek-R1 ( Guo et al., 2025 ) trained with 140x more compute (1152 vs 8 H100 hours). Results show that Qwen2.5-Math- CFT reaches the same average performance across 5 math benchmarks, highlighting the efficiency and effectiveness of CFT for reasoning-focused tasks. To understand the models’ general instruction following abilities, we also evaluate our Qwen2.5-Math-CFT on MT-Bench ( Zheng et al., 2023 ) and IF_Eval ( Zhou et al., 2023 ) . We show that our method also enhances general instruction-following abilities. It outperforms our SFTed version and the official Qwen2.5-Math-Instruct by a notable margin. To better understand different factors of CFT , we conducted comprehensive ablation studies: 1. Robustness to dataset sources: Comparing WebInstruct ( Yue et al., 2024b ) against MetaMathQA ( Yu et al., 2024 ) and NuminaMath ( Li et al., 2024b ) , we observed that WebInstruct provided a slight advantage (3%+) due to its diversity and broader topic coverage. 2. Robustness to noisy response sources: Experiments with original noisy responses and Qwen2.5-base responses critiqued by GPT-4o showed negligible performance differences. 3. Flexibility to the teacher: Using a weaker critique dataset synthesized by GPT-4o-mini still yielded notable improvements over SFT, despite a 4% overall score drop. 4. Controlling for token length: Even controlling token length, shorter critique examples significantly outperformed standard SFT (55.2% vs. 50.4%), confirming improvements stem from critique-based training rather than increased sequence length. Our approach not only demonstrates strong improvement on reasoning-focused tasks, but also exhibits notable improvement over general-purpose instruction following tasks. These evidence has shown great potential of CFT to replace SFT

Read the complete text