Seed-Coder: Let the Code Model Curate Data for Itself

ByteDance Seed Homepage GitHub Hugging Face · 2025

Read Seed-Coder: Let the Code Model Curate Data for Itself by ByteDance Seed Homepage GitHub Hugging Face (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03524.txt.

Introduction

The ByteDance Seed team lets a code model curate its own training data in a 2025 paper. Code data lifts not just coding skill but general smarts. Human filtering rules are costly and hard to scale. Their pipeline uses models to score and filter code instead.

The result is Seed-Coder, a family of open 8B models. Base, instruct, and reasoning versions share the data recipe. Fine-tuning and preference tuning polish the instruct model. Long chain-of-thought training powers the reasoning version. Scores top open models of similar size across code tasks. Some much larger models are beaten as well. Editing, completion, and reasoning tasks all shine.

Catalog record

CREATORByteDance Seed Homepage GitHub Hugging Face
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seed-Coder: Let the Code Model Curate Data for Itself ByteDance Seed Homepage GitHub Hugging Face Abstract Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks. Figure 1 : Benchmark performance of instruct and reasoning variants of Seed-Coder-8B. 1 Introduction Large language models (LLMs) have achieved superior performance in a wide range of coding tasks, showing their potential to revolutionize the entire ecosystem of software development. Current state-of-the-art models, such as Claude 3.7 Sonnet [ Anthropic 2025 ] and OpenAI o3 [ OpenAI 2025 ] , have demonstrated unprecedented capability in a wide range of code-related tasks, including code generation, code explanation, code debugging, code editing, and real-world software engineering tasks. These models can largely enhance developers’ productivity and provide substantial support for the software industry, showing the potential to gradually automate the process of software development. However, these cutting-edge models remain proprietary and have disclosed very little about their training data. State-of-the-art open-source LLMs, such as DeepSeek-R1 [ DeepSeek-AI et al. 2025 ] and Qwen2.5-Coder [ Hui et al. 2024 ] , have invigorated the research community by offering self-deployable models that demonstrate competitive performance on coding tasks. While their technical reports provide valuable insights into both pretraining and post-training techniques, many offer only a high-level overview of their data processing methodologies. Among these ideas, one broad consensus for data quality control lies in that leveraging human expertise to curate and refine datasets is an intuitive and effective approach to enhancing the quality of code pretraining data. For example, DeepSeek-Coder [ Guo et al. 2024 ] and DeepSeek-Coder-V2 [ DeepSeek-AI et al. 2024a ] apply a set of filtering rules as the initial data processing step, following the filtering rules used in StarCoder [ Li et al. 2023 ] . Qwen2.5-Coder [ Hui et al. 2024 ] also utilizes a series of rule-based filtering methods similar to DeepSeek-Coder. Moreover, OpenCoder [ Huang et al. 2025 ] incorporates over 130 hand-crafted filtering rules with customized weights in their pretraining data pipeline. However, hand-crafted rules are prone to conflicts among themselves and may incur high costs to extend and maintain across diverse programming languages. Existing human-centric data filtering approaches that heavily rely on hand-crafted rules often introduce subjective biases and inherently lack scalability. We believe that these limitations align with insights from the widely cited essay “The Bitter Lesson” [ Sutton 2019 ] : AI researchers often favor human-centric methods due to their short-term advantages, yet these approaches inevitably plateau and even inhibit long-term progress; the breakthrough progress eventually arrives by the opposing approach via scaling computation and data. In the context of code pretraining data, the human-centric approaches appear particularly favorable, since many AI researchers are skilled programmers themselves and feel confident in evaluating code quality. Nevertheless, human-centric methods would ultimately prove restrictive and tend to impede advancements of code LLMs in the long run. In this report, we present a model-centric data pipeline that minimizes human involvement in constructing code pretraining data. Our data pipeline predominantly leverages LLMs, instead of hand-crafted rules, for scoring and filtering code data. Compared to human-centric approaches, the use of LLMs is particularly advantageous, as they effectively capture nuanced standards of code quality that are difficult to quantify explicitly, and simultaneously provide scalable, self-contained evaluations capable of processing billions of samples consistently. We applied these LLM filters during the preprocessing of our data – including GitHub code, GitHub commits, and code-related web data – and curated a code pretraining corpus comprising a total of 6 trillion tokens. Building on this foundation, we introduce Seed-Coder, a family of state-of-the-art open-source code LLMs at the 8B scale, including a base model, an instruct model, and a reasoning model. Starting from the base model trained with the proposed pretraining pipeline, the instruct model is further fine-tuned on large-scale synthetic data generated and filtered by LLMs, followed by direct preference optimization (DPO) to enhance the instruction-following capabilities. Meanwhile, the reasoning model improves multi-step reasoning in complex coding tasks by applying Long-Chain-of-Thought (LongCoT) reinforcement learning. Extensive evaluations on well-established benchmarks on a diverse set of coding tasks were conducted to showcase the advanced capabilities of the three variants of Seed-Coder. We release this lightweight yet powerful model family with the intention of providing research insights while maintaining a manageable scale to facilitate further exploration within the open-source community. 2 Pretraining 2.1 Data Pipeline Figure 2 : Processing pipeline for pretraining data. We collected data from GitHub and web archives. The raw data were processed into four categories: file-level codes (yellow), repository-level codes (blue), GitHub commits (red) and code-related web data (green). For each phase in pretraining, we combined and reorganized the processed data from the four categories, indicated by colors on the top-right of the blocks. Producing data for LLM pretraining typically requires complex data pipelines and intricate data lineage. To ensure efficiency and reduce computation and storage costs, we propose a parallel design, which decouples the sequential dependency of various filters so as to streamline our data pipeline. As illustrated in Figure 2 , our end-to-end data pipeline takes as input the raw code data crawled from GitHub and web archives, and outputs the final pretraining data after a few steps of data processing. All the preprocessing and filtering modules were disentangled so that each can run individually to avoid re-running the entire pipeline for better support of incremental data expansion and flexible pipeline manipulation. Our data pipeline starts with the standard exact- and near-deduplication to get a holistic set of code-related files without duplication. Then we applied basic filters with minimal rules such as language inferring to get rid of irrelevant and non-code data. On top of the preprocessed data, we developed advanced quality filters powered by LLMs, which capture general standards and exhibit strong generalizability to address the wide variety across the large-scale data. Our filtered data falls into four categories: • File-level codes: Individual code files from GitHub Data. • Repository-level codes: Code files structured based on the repositories. • Commits data: Snapshots of GitHub Commits, including commit messages, repository metadata, all relevant files, and code patches. • Code-related web data: Documents from web archives that contains code

Read the complete text