Critique-Guided Distillation: Improving Supervised Fine-tuning via Better Distillation
Berkcan Kapusuzoglu, Supriyo Chakraborty, Chia-Hsuan Lee, Sambit Sahu · 2025
Read Critique-Guided Distillation: Improving Supervised Fine-tuning via Better Distillation by Berkcan Kapusuzoglu, Supriyo Chakraborty, Chia-Hsuan Lee, Sambit Sahu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11628.txt.
Introduction
Berkcan Kapusuzoglu and co-authors present a 2025 training method for language models that learn from expert examples. Models trained this way often copy the surface of good answers without taking in the thinking behind them. The team's method, called critique-guided distillation in the catalog title, trains the model to fix flawed answers while a teacher's written critiques point out the errors.
The critiques appear only during training, never when the model is used, so the model must take the error checking habit into itself. Tests across five model families showed steady gains on math reasoning tasks over standard training, with large gains on some contest problem sets. The method also kept general instruction following intact, where a rival approach that trains models to write critiques caused clear losses.
Catalog record
| CREATOR | Berkcan Kapusuzoglu, Supriyo Chakraborty, Chia-Hsuan Lee, Sambit Sahu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Critique-Guided Distillation: Improving Supervised Fine-tuning via Better Distillation Berkcan Kapusuzoglu, Supriyo Chakraborty, Chia-Hsuan Lee, Sambit Sahu Abstract Supervised fine-tuning (SFT) using expert demonstrations often suffer from the imitation problem, where the model learns to reproduce the correct responses without understanding the underlying rationale. To address this limitation, we propose Critique-Guided Distillation (CGD) , a novel multi-stage framework that integrates teacher model generated explanatory critiques and refined responses into the SFT process. A student model is then trained to map the triplet of prompt, teacher critique, and its own initial response to the corresponding refined teacher response, thereby learning both what to imitate and why . Using entropy-based analysis, we show that CGD reduces refinement uncertainty and can be interpreted as a Bayesian posterior update. We perform extensive empirical evaluation of CGD , on variety of benchmark tasks, and demonstrate significant gains on both math (AMC23 +17.5%) and language understanding tasks (MMLU-Pro +6.3%), while successfully mitigating the format drift issues observed in previous critique fine-tuning (CFT) techniques. 1 Introduction Supervised fine-tuning (SFT) is a foundational technique for teaching large language models (LLMs) to perform diverse downstream tasks by mimicking expert-annotated outputs [ 39 , 27 ] . Despite its success, vanilla SFT has notable limitations: it increases model’s tendency to hallucinate [ 12 ] , exhibits limited out-of-distribution generalization [ 5 ] , and struggles to generalize to harder problem instances [ 33 , 32 ] . These shortcomings raise fundamental questions about SFT’s capacity for robust, and complex reasoning. An alternative approach to improve reasoning leverages critique and revision at inference time: a model generates an initial answer, critiques it, then refines its output based on that critique [ 16 , 22 , 29 , 28 ] . While effective, these multi-pass prompting methods incur high computational cost and latency during deployment. To integrate critique signals without extra inference costs, recent works has moved these steps into training. Rejection Sampling Fine-Tuning (RFT) [ 45 ] trains the model on its own generated outputs that are verified or ranked by a reward model, thus incorporating value-based feedback. Critique Fine-Tuning (CFT) instead trains a student model to reproduce teacher-generated critiques [ 37 ] . Although CFT outperforms vanilla SFT on several math benchmarks, prolonged CFT can induce output-format drift—overfitting to critique patterns rather than stable answer structures and its gains are sensitive to the quality of the critiques provided. In this work, we introduce Critique-Guided Distillation (CGD) (Fig. 1 ), a novel multi-step fine-tuning paradigm in which a student model learns to transform its own initial outputs into high-quality refinements—rather than merely generating critiques. Concretely, we condition a teacher model to produce both critiques and corresponding corrected answers, and train the student to map its raw response to the teacher’s refined version by conditioning it on the critique. By internalizing not only how to identify errors but what a polished response looks like, CGD closes the loop between diagnosis and correction, yielding more robust formatting comparing to CFT as shown in Fig. 2 , and significantly lower inference overhead than multi-pass critique methods. Figure 1 : Overview of Critique-Guided Distillation (CGD) . The student model first generates an initial response. The teacher model then produces targeted critiques and a corresponding refined answer. Finally, we supervise-finetune the student model to generate the refined answer given the prompt and the critiques generated by the teacher model. Figure 2 : Comparing Supervised fine-tuning (SFT), Critique Fine-Tuning (CFT) and Critique-Guided Distillation (CGD) . By conditioning answer generation on the critique, CGD avoids format drift (the model continues to generate answers, not critiques) and ensures feedback is explicit and grounded. From a probabilistic perspective, each critique acts as evidence in a Bayesian update, the original model distribution P(y|x) (where y is an answer and x a query) is refined to a posterior P(y|x,c)\propto P(c|x,y)P(y|x) when conditioning on the critique c . By the properties of conditional entropy, adding this evidence guarantees a reduction in uncertainty under critique guidance. Thus, CGD ’s distillation systematically reduces the model’s uncertainty in line with the critique guidance. We empirically validate CGD on mathematical reasoning and broad knowledge benchmarks, observing +17.5% and +15.0% absolute accuracy gains over SFT and CFT respectively on the challenging AMC23 dataset. Similarly, on OlympiadBench, CGD achieves a +12.9% gain over SFT and +8.0% over CFT, further reinforcing its effectiveness on hard math reasoning. Figure 3 visualizes this performance trend, showing that CGD consistently improves over strong baselines like Distilled SFT and CFT across all evaluation tasks. These gains are particularly notable given that all methods are trained on the same 100K WebInstruct samples with access to the same teacher model (LLaMA3.3-70B Instruct), suggesting that it is the structured use of critiques—not just data quantity or teacher strength—that drives performance. This supports our hypothesis that critique-conditioned refinement helps student models internalize not just what to imitate but why, yielding more robust and generalizable reasoning behavior. Figure 3 : Performance comparison of CGD , using a LLaMA3.3-70B Instruct teacher model to generate critiques and refined answers, with 100K samples from WebInstruct [ 46 ] . The LLaMA3.1-8B Instruct student model is trained using both the critique and the initial answer as input, and the refined answer as the target. Baselines include Distilled SFT, which uses soft distillation using the same teacher (LLaMA3.3-70B Instruct) model on the same WebInstruct data, and CFT, which trains on GPT-4o-generated critiques [ 37 ] . Our contributions are as follows : • We introduce CGD , a critique-conditioned distillation framework that preserves answer-generation format and integrates explanatory critiques. • We provide a theoretical analysis and a Bayesian interpretation of CGD , showing that each critique acts as evidence that narrows the model’s output distribution (reducing entropy), thus formalizing why CGD produces more confident, accurate predictions. • We empirically validate CGD ’s effectiveness on mathematical reasoning and knowledge benchmarks, showing significant gains over SFT and CFT while mitigating format drift. We report all our results using a LLaMA3.1-8B Instruct student model, and LLaMA3.3-70B Instruct teacher model. 2 Related Work We organize prior work under the following broad categories. Supervised Fine-Tuning Limitations Standard supervised fine-tuning (SFT) trains LLMs to mimic expert demonstrations, but it often induces the imitation problem, where models reproduce outputs without internalizing reasoning processes. Previous study shows that fine-tuning language model on new knowledge increases model’s tendency to hallucinate [ 12 ] . Furthermore, fine-tuned models exhibit poor out-of-distribution performance [ 5 ] , and gains on familiar data often come at the cost of reliability on unseen distributions [ 18 ] . Recent work has also shown that SFT on reasoning trajectories can substantially boost mathematical problem-solving with only a few thousand examples [ 23 , 42 ] . Nonetheless, vanilla SFT still struggles to generalize to harder problem instances, leaving open the question of its limits on complex reasoning [ 33 , 32 ] . These limitations motivate integrating critique-and-correction mechanisms beyond naive answer imitation to achieve more robust reasoning and improved downstream performance. Self-Correction and Critiques Prior work has shown that LLMs can critique their own outputs and then refine those outputs based on self-generated feedback, yielding stronger reasoning and improved downstream performance [ 16 , 22 , 28 ]