Self-Taught Self-Correction for Small Language Models
Viktor Moskvoretskii, Chris Biemann, Irina Nikishina · 2025
Read Self-Taught Self-Correction for Small Language Models by Viktor Moskvoretskii, Chris Biemann, Irina Nikishina (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08681.txt.
Introduction
Viktor Moskvoretskii, Chris Biemann, and Irina Nikishina published a paper in 2025 on teaching small language models to fix their own mistakes, using only their own output as training data. Big models are error-prone too, but self-correction research leaned on outside tools or giant proprietary teachers. Small models, cheap and private, had been left out.
Their algorithm, STaSC, fine-tunes a model in rounds on corrections it generates itself, with several design choices tuned along the way. On a question-answering task the model genuinely learned to revise, with significant accuracy gains over its uncorrected self. The analysis dissects which choices drive the learning and how the correction skill develops. Code and the lightweight models are released for others to extend.
Catalog record
| CREATOR | Viktor Moskvoretskii, Chris Biemann, Irina Nikishina |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08681
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Taught Self-Correction for Small Language Models Viktor Moskvoretskii, Chris Biemann, Irina Nikishina Abstract Although large language models (LLMs) have achieved remarkable performance across various tasks, they remain prone to errors. A key challenge is enabling them to self-correct. While prior research has relied on external tools or large proprietary models, this work explores self-correction in small language models (SLMs) through iterative fine-tuning using solely self-generated data. We introduce the Self-Taught Self-Correction (STaSC) algorithm, which incorporates multiple algorithmic design choices. Experimental results on a question-answering task demonstrate that STaSC effectively learns self-correction, leading to significant performance improvements. Our analysis further provides insights into the mechanisms of self-correction and the impact of different design choices on learning dynamics and overall performance. To support future research, we release our user-friendly codebase and lightweight models. \Exlabelwidth 1 Introduction Recent advanced LLM employ complex reasoning ( Guo et al. 2025 ) and meta-reasoning ( Xiang et al. 2025 ) , expanding their capabilities. However, even the most advanced models are prone to errors, including hallucinations ( Huang et al. 2025 ) and logical inconsistencies ( Ghosh et al. 2024 ) , requiring external verification or human intervention. To address those problems, self-correction — the ability to revise their own outputs — has been evolved ( Madaan et al. 2023 ) . The existing approaches mostly use zero-shot prompting ( Madaan et al. 2023 ; Shinn et al. 2024 ) , external evaluators for correction or feedback ( Zhang et al. 2024 ) or apply large proprietary models and focus specifically on mathematical tasks ( Kumar et al. 2024 ) . In this study, we focus on self-correction without external information or evaluators, ensuring inference efficiency while relying solely on the model’s intrinsic knowledge. We investigate the self-correction capabilities of small language models (SLMs) by applying iterative fine-tuning on self-generated data, allowing models to improve their outputs without external supervision. We introduce the Self-Taught Self-Correction (STaSC) algorithm, which trains models to self-correct using self-generated trajectories, adapting the core idea of STaR ( Zelikman et al. 2022 ) . STaSC extends and unifies approaches that iteratively train with self-generated trajectories, including the Self-Correction (SC) algorithm ( Welleck et al. 2022 ) , by incorporating flexible design choices. Unlike prior methods, STaSC provides control over initial answer exploration, correction filtering, and iterative fine-tuning, encompassing SC as a special case and demonstrating how different algorithmic choices impact self-correction performance. Figure 1: Illustration of the self-improvement method STaR (left) Zelikman et al. 2022 , self-correction method SC (center) Welleck et al. 2022 , and our method, STaSC (right). STaCS offers flexible control over initial answer exploration, correction filtering, and iterative fine-tuning. It is inspired by STaR and effectively encompasses SC as a special case. SC and STaSC allow several initial answers and corrections. The dotted line in the STaSC denotes two possible setups: fine-tuning the model and generating from it at the next iteration (Evolving Fine-Tuning) and keeping the Generator frozen and fine-tuning the Corrector model only (Fixed Fine-Tuning). Our results on the Natural Questions dataset ( Kwiatkowski et al. 2019 ) show that SLMs can learn to self-correct using self-synthesized data, while also improving their initial answer quality despite being trained solely for corrections. We release easy-to-use and adaptable code for self-correction algorithms at https://github.com/VityaVitalich/STASC . The contributions of the paper are as follows: • We propose the Self-Taught Self-Correction (STaSC) algorithm, unifying and extending existing self-correction methods trained with self-generated trajectories. • We conduct extensive experiments on a purely Natural Language Processing (NLP) task — Question Answering — using an open-source SLMs, demonstrating their ability to learn self-correction with self-synthesized data. • We release open-source, easily adaptable code for self-correction, along with efficient SLMs with fewer than 4B parameters, making self-correction practical and accessible. 2 Self-Taught Self-Correction In this section, we introduce the Self-Taught Self-Correction (STaSC) algorithm, an adaptation of STaR ( Zelikman et al. 2022 ) for self-correction through iterative fine-tuning on self-synthesized data. STaSC unifies and extends various self-correction approaches using self-synthesized data, including the Self-Correction (SC) algorithm ( Welleck et al. 2022 ) . 2.1 Foundations and Enhancements Figure 1 presents three algorithms highlighting their similarities and dissimilarities. The left part describes the original STaR algorithm which focuses on generating reasoning paths, filtering correct ones, and fine-tuning them accordingly. The central part of the Figure 1 demonstrates the Self-Correction (SC) algorithm which fine-tunes the corrector model on corrections leading to improvement, keeping the initial Generator model fixed. The right part refers to STaSC , adapting the idea of STaR of iteratively refining model outputs by sampling an initial answer, generating corrections, filtering correct trajectories, and fine-tuning on the successful revisions. This process is repeated over multiple iterations, progressively enhancing the model’s accuracy. The key extension of our algorithm over previous baselines is the incorporation of flexible algorithmic choices. Initial answers can be drawn from either a frozen model or the previous iteration, corrections can be filtered based on strict improvements or by allowing non-worsening revisions, and fine-tuning can be performed from either the initial model or the latest iteration. 2.2 Formal Definition and Algorithmic Choices Below, we outline the Algorithm 1 steps and key design choices, each of which influences the self-correction process. For each choice, we define an abbreviation that will be used to denote specific algorithm configurations, such as STaSC FIF , where subscripts specify the selected options. 2.2.1 Requirements and Notation The algorithm begins with an initial language model state, M_{0} , and an initial dataset, D_{0} , consisting of input-output pairs (x,y) . Additionally, we define a number of improvement iterations N , the number of sampled initial generations N_{\text{init}} , the number of sampled corrections N_{\text{corr}} , and a reward function r , which evaluates the quality of model-generated outputs. Algorithm 1 Self-Taught Self-Correction (STaSC) 1: Initial model M_{0} , dataset D_{0} , number of iterations N , initial samples N_{\text{init}} , correction samples N_{\text{corr}} , reward function r 2: for n=1 to N do 3: Step 1: Sample Initial Answers 4: \hat{Y}^{1}_{i}=\{\hat{y}_{ij}^{1}\}_{j=1}^{N_{\text{init}}}\sim M(x_{i}) , \forall x_{i}\in D_{0} 5: Option 1: M=M_{n-1} (Evolving Initialization) 6: Option 2: M=M_{0} (Fixed Initialization) 7: Step 2: Sample Corrections 8: \hat{Y}^{2}_{i}=\{\hat{y}_{ijk}^{2}\}_{k=1}^{N_{\text{corr}}}\sim M_{n-1}(x_{i},\hat{y}_{ij}^{1}) , \forall\hat{y}_{ij}^{1}\in\hat{Y}^{1}_{i} 9: Step 3: Filter Corrections 10: D_{n}^{+}=\{(x_{i},\hat{y}_{ij}^{1},\hat{y}_{ijk}^{2})\;|\;r(\hat{y}_{ijk}^{2})>r(\hat{y}_{ij}^{1})\} 11: D_{n}^{=}=\{(x_{i},\hat{y}_{ij}^{1},\hat{y}_{ijk}^{2})\;|\;r(\hat{y}_{ijk}^{2})=r(\hat{y}_{ij}^{1})\geq t\} 12: Option 1: D_{n}=D_{n}^{+} (Improving Filter) 13: Option 2: D_{n}=D_{n}^{+}\cup D_{n}^{=} (Non-Decreasing Filter) 14: Step 4: Fine-Tuning 15: M_{n}=\text{train}(M,\{\hat{y}_{ijk}^{2}\;|\;(x_{i},\hat{y}_{ij}^{1},\hat{y}_{ijk}^{2})\in D_{n}\}) 16: Option 1: M=M_{0} (Fixed Fine-Tuning) 17: Option 2: M=M_{n-1} (Evolvong Fine-Tuning) 18: end for Step 1: Sampling Initial Answers In the first step, we sample N_{\text{init}} initial answers \hat{y}^{1} for each input x in the dataset D_{0} . The primary design choice here is the selection of the model M used for sampling: • Fixed Initialization (STaSC F** ) – Initial answers are sampled using the frozen model M_{0} , as in the SC algorithm ( Welleck et al. 2022 ) . • Evolving Initialization (STaSC E** ) – Initial answers are sampled using the model from the previous iteration M_{n-1} , as in the original STaR ( Zelikman et al. 2022 ) . Fixed Initialization ensures robustness to variations in self-improvement and dataset shifts, maintaining consistency across iterations. In contrast, Evolving Initialization allows for greater exploration, potentially leading