SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs

Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, Mengdi Zhang · 2025

Read SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs by Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, Mengdi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07858.txt.

Introduction

Training data for code-writing language models can be made by small open models themselves, argue Xinyu Zhang and 7 co-authors in a 2025 paper. Large code models usually learn from examples distilled from costly closed systems. The team instead bootstraps a small model of about seven billion parameters into a data synthesizer, a maker of practice examples.

Their method, SCoder, works in rounds. A few excellent synthesis examples from a strong model first teach the small model how to create good training data. Repeated self-distillation then sharpens that skill, so the model generates better code instructions on its own. Cheaper, self-made data could let open code models improve without depending on proprietary teachers.

Catalog record

CREATORXinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, Mengdi Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, Mengdi Zhang Abstract Existing code large language models (LLMs) often rely on large-scale instruction data distilled from proprietary LLMs for fine-tuning, which typically incurs high costs. In this paper, we explore the potential of small-scale open-source LLMs (e.g., 7B) as synthesizers for high-quality code instruction data construction. We first observe that the data synthesis capability of small-scale LLMs can be enhanced by training on a few superior data synthesis samples from proprietary LLMs. Building on this, we propose a novel iterative self-distillation approach to bootstrap small-scale LLMs, transforming them into powerful synthesizers that reduce reliance on proprietary LLMs and minimize costs. Concretely, in each iteration, to obtain diverse and high-quality self-distilled data, we design multi-checkpoint sampling and multi-aspect scoring strategies for initial data selection. Furthermore, to identify the most influential samples, we introduce a gradient-based influence estimation method for final data filtering. Based on the code instruction datasets from the small-scale synthesizers, we develop SCoder, a family of code generation models fine-tuned from DeepSeek-Coder. SCoder models achieve state-of-the-art code generation capabilities, demonstrating the effectiveness of our method. 1 Introduction Code generation has long been a central challenge in computer science and has attracted wide attention from the research community. Recent advancements in code large language models (LLMs) Chen et al. 2021 ; Li et al. 2022 ; Li et al. 2023 ; Chowdhery et al. 2023 ; Rozière et al. 2023 ; Lozhkov et al. 2024 have led to significant breakthroughs. These models can generate code that closely aligns with user intent and are increasingly being widely adopted. Typically, instruction tuning on base models (e.g., DeepSeek-Coder-Base) is a crucial step in developing high-performance code LLMs. Therefore, extensive research on code LLMs focuses on constructing high-quality instruction data. A common approach involves distilling knowledge from proprietary LLMs. For instance, Code Alpaca Chaudhary 2023 and WizardCoder Luo et al. 2024 are fine-tuned with instruction data distilled from GPT-3.5, using Self-Instruct Wang et al. 2023 and Evol-Instruct Xu et al. 2024 , respectively. Additionally, MagicoderS Wei et al. 2024 is fine-tuned on data distilled from both GPT-3.5 and GPT-4, using OSS-Instruct to generate coding problems and solutions based on the given code snippets. While these methods have proven effective, they all suffer from the cost-intensive issue caused by the distillation of large-scale instruction data from the proprietary LLMs like GPT-3.5 and GPT-4. Figure 1: Left : The performance of code generation models on HumanEval using data provided by different synthesizers (Qwen2.5-Coder-7B or -14B). Right : The performance of our SCoder and the baseline. SCoder uses 60K instruction data generated by a small-scale synthesizer, and the baseline uses 75K instruction data generated by proprietary LLMs. All code generation models are fine-tuned from DeepSeek-Coder-6.7B-Base. In this paper, we explore the potential of relatively small-scale (7B, 8B, and 14B) open-source LLMs as synthesizers for code instruction data construction. Previous works have shown that small LLMs can assist in pre-training data synthesis for non-code domains Yang et al. 2024 . However, instruction data typically takes a different form from pre-training data and requires higher quality standards Wang et al. 2025 . To validate the feasibility of small LLMs in synthesizing code instruction data, we conduct a preliminary experiment. First, we use small-scale LLMs as original synthesizers and further train them on a limited set of proprietary LLM-distilled samples as enhanced synthesizers. Then, we fine-tune code generation models using data provided by them. The results on the left of Figure 1 show that the instruction data provided by the enhanced synthesizer outperforms that of the original, highlighting that a few superior samples can unleash the data synthesis potential of small models. However, distilling more proprietary samples to further improve the synthesis capability of small synthesizers would again trigger the cost-intensive issue. Therefore, a crucial question arises: Can we continuously improve the data synthesis capability of small-scale synthesizers without relying on proprietary LLMs’ samples? To address this, we propose a progressive self-distillation method that iteratively bootstraps the code instruction data synthesis capability of small-scale LLMs. Specifically, starting with an enhanced synthesizer, we employ a two-step approach in each iteration to obtain high-quality self-distilled data synthesis samples for further training. First, we design multi-checkpoint sampling and multi-aspect scoring strategies to obtain diverse and reliable self-distilled samples. Then, we introduce a gradient-based influence estimation method to further select the influential ones by comparing the gradients induced by self-distilled samples with those induced by superior samples from proprietary LLMs. We validate our method on small-scale LLMs like Qwen2.5-Coder-7B/14B-Ins Hui et al. 2024 , improving their data synthesis capabilities as shown in the left of Figure 1 , and transforming them into powerful data synthesizers. Based on the code instruction datasets provided by our small-scale synthesizers, we develop SCoder, a family of code generation models fine-tuned from DeepSeek-Coder-6.7B-Base Guo et al. 2024 . Experimental results on HumanEval (+) Chen et al. 2021 ; Liu et al. 2023 , MBPP (+) Austin et al. 2021 , LiveCodeBench Jain et al. 2024 , and BigCodeBench Zhuo et al. 2024 show that SCoder outperforms or matches state-of-the-art code LLMs that use the instruction data from proprietary LLMs. Overall, our contributions can be summarized as follows: • We propose a novel iterative self-distillation approach that transforms small-scale LLMs into effective synthesizers of code instruction data. Using the instruction data generated by these synthesizers, we train a family of code generation models (SCoder), which achieve performance comparable to that of models relying on proprietary LLM-distilled data. • To obtain diverse and high-quality self-distilled data, we design multi-checkpoint sampling and multi-aspect scoring strategies for initial data selection. To further identify the most influential samples, we introduce a gradient-based influence estimation method for final data filtering. • We fine-tune the code generation models (SCoder) based on the datasets generated by our small-scale synthesizers. Experimental results on multiple benchmarks show the effectiveness of our method. 2 Related Work 2.1 Code Large Language Models In recent years, large language models have garnered unprecedented attention Zhang et al. 2024 ; Azaria et al. 2024 ; Chen 2024 ; Zhou et al. 2025 , and LLM-driven code generation has achieved remarkable progress. Prominent closed-source models such as Codex Chen et al. 2021 , GPT-4 OpenAI 2023 , PaLM Chowdhery et al. 2023 , and Gemini Anil et al. 2023 have shown impressive performance across various code generation benchmarks. Meanwhile, open-source models like CodeGen Nijkamp et al. 2023 , CodeGeeX Zheng et al. 2023 , StarCoder Li et al. 2023 , CodeLlama Rozière et al. 2023 , DeepSeek-Coder Guo et al. 2024 , and CodeQwen Hui et al. 2024 have also made substantial contributions. These models not only enhance code generation capabilities but also promote more efficient and automated software development. Typically, such models are developed through continual pre-training Rozière et al. 2023 , followed by supervised fine-tuning (SFT) Yu et al. 2023 . While pre-training utilizes large-scale, unannotated code corpora, SFT relies on high-quality labeled instruction data, whose construction remains a key challenge Ding et al. 2024 . 2.2 Code Instruction Data Synthesis Creating diverse and complex code

Read the complete text