CodeEvo : Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback

Qiushi Sun Jinyang Gong Lei Li, Qipeng Guo Fei Yuan · 2025

Read CodeEvo : Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback by Qiushi Sun Jinyang Gong Lei Li, Qipeng Guo Fei Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22080.txt.

Introduction

Sun and co-authors grow in 2025 training data for code writing models by letting two agents argue their way to better examples. Hand made instruction code pairs are costly, and machine made ones often rest on rigid recipes that flatten the logic. Their Coder writes solutions while a Reviewer steers the whole path.

The Reviewer drafts a schema that plans difficulty and structure before code appears. Compiler verdicts plus semantic checks verify each pair in the loop. The resulting hundred thousand pairs carry stepped difficulty. Models tuned on them beat baselines across coding benchmarks, the team report.

Catalog record

CREATORQiushi Sun Jinyang Gong Lei Li, Qipeng Guo Fei Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CodeEvo : Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback Qiushi Sun Jinyang Gong Lei Li, Qipeng Guo Fei Yuan Abstract Acquiring high-quality instruction-code pairs is essential for training Large Language Models (LLMs) for code generation. Manually curated data is expensive and inherently limited in scale, motivating the development of code-centric synthesis methods. Yet, current approaches either focus on augmenting existing code or rely on predefined heuristics, both lacking rigorous data validation, which results in synthetic data that is ungrounded, repetitive, or overly simplistic. Inspired by collaborative programming practices, we propose CodeEvo , a framework that synthesizes code data through iterative interactions between two LLM agents: a Coder , which generates candidate code and test cases based on given instructions, and a Reviewer , which guides the synthesis process by producing new instructions and feedback. We further introduce a hybrid feedback mechanism that combines compiler determinism with the generative flexibility of agents, enabling automatic quality control throughout synthesis. Extensive experiments demonstrate that models fine-tuned on CodeEvo data significantly outperform established baselines across code generation benchmarks with various difficulties. In-depth analyses further provide insights from multiple perspectives into effective code-centric data synthesis. \NewTableCommand \seprule \NewTableCommand \uniquerule 1 Introduction Figure 1: Comparison of synthesized code data quality across human examples and code produced by Qwen2.5-Coder-32B, DeepSeek-V3, and Llama-3.1-8B. The rapid development of Large Language Models (LLMs) has significantly advanced code intelligence ( Sun et al. 2024a ) , powering applications ranging from line-level code completion to competition-level problem solving. To further enhance their performance on code generation, it is essential to train these models with complex, diverse, and grounded instruction-code pairs ( Zan et al. 2023 ) . While manually curated data serve as ideal resources, their collection is labor-intensive, difficult to scale, and gradually exhausted ( Huang et al. 2024b ) . These limitations have stimulated growing interest in constructing code-centric synthetic data with minimal human intervention. After early attempts that leverage symbolic augmentation over existing code references ( Wang et al. 2017 ; Feng et al. 2018 ) , recent research has shifted toward using LLMs to automatically generate instruction-code pairs. These methods, such as Self-Instruct ( Wang et al. 2023 ) and Evol-Instruct ( Luo et al. 2024b ) , aim to bootstrap massive data using powerful models and predefined heuristics. While these approaches enable data construction, they often fall short in ensuring semantic correctness and executability ( Liu et al. 2023 ) . As shown in Figure 1 , we sample instruction-code pairs synthesized by various (Code)LLMs using Evol-Instruct heuristics ( Xu et al. 2024a ) . Many of these samples fail to execute or do not pass the provided unit tests, indicating substantial quality gaps. These shortcomings can be attributed to two main factors: (1) instructions are often poorly grounded, leading to vague or inconsistent objectives; and (2) generated codes lack proper validation, due to the absence of robust mechanisms to enforce correctness during synthesis. This motivates a key question: Can we design a fully automated and reference-free synthesis pipeline that produces well-grounded and executable instruction-code pairs? Recently emerging LLM agents have demonstrated strong interactive capabilities ( Sun et al. 2023 ) , enabling them to perform tasks through multi-turn interactions and feedback-driven decision making ( e.g. , collaborative programming; Qian et al. 2024 ; Wang et al. 2025a ). These make them promising candidates for moving beyond vanilla data generation toward verifiable and adaptive synthesis pipelines. Inspired by this potential, we propose CodeEvo , an interaction-driven synthesis framework that orchestrates LLM agents to generate high-quality code-centric data. Specifically, a Coder agent produces candidate code and tests based on given instructions, while a Reviewer agent provides tailored feedback and dynamically constructs new instructions iteratively. To address the two core challenges in instruction-code synthesis, CodeEvo incorporates two key mechanisms: (1) To lift instruction quality, we exploit keyword usage to construct grounded instructions, where Reviewer conditions its generation on task-specific keywords to facilitate progressively challenging, targeted problems. (2) To boost functional correctness of code, we design a hybrid feedback mechanism that integrates “compiler-in-the-loop” verification with LLM-based evaluations. By combining the determinism of program execution with the flexibility of LLM judgment, it substantially reduces the generation of erroneous code solutions while retaining adaptability. The entire pipeline operates with only a small set of seed instructions as input, and requires no human annotation or gold references , all while being driven by accessible, medium-sized models. Experiments across multiple backbones and benchmarks demonstrate that CodeEvo significantly outperforms established data synthesis methods. Remarkably, CodeEvo achieves better performance than competing approaches using several times more data, indicating the superiority of our targeted, feedback-driven synthesis over sheer data volume. Our primary contributions are as follows: • We propose CodeEvo , an interaction-driven synthesis framework that systematically lifts the quality of code-centric synthetic data from both the instruction and code perspectives. • We introduce an innovative hybrid feedback mechanism that effectively combines the determinism of compiler verification with the generative flexibility of LLM agents. • Through extensive experiments and analysis, we share insights into the key attributes, including quality, diversity, scalability, and difficulty of synthetic code data. Figure 2: Overview of the CodeEvo framework. The synthesis process begins with seed instructions from different sources. Through continuous interaction, the Coder and Reviewer agents collaboratively construct trajectories of instruction, solution, validation, and refinement. The loop marked with \rightarrow illustrates the flow of data synthesis, where new instructions are derived and paired with validated code. The loop marked with \rightarrow captures the validation cycle, incorporating natural language and compiler feedback to guide refinement. Instruction-code pairs are extracted from the validated trajectories and used for downstream model training. 2 Related Works LLM-based Agents Interaction. The interactive capabilities of language agents ( Sumers et al. 2024 ) , whether with other agents or the environment, have garnered significant attention ( Park et al. 2023 ) . These interactions allow for complex problem-solving approaches such as collaboration ( Sun et al. 2023 ; Hong et al. 2024 ) or role-playing ( Li et al. 2023 ; Islam et al. 2024 ) , which have proven effective in diverse applications like software engineering ( Qian et al. 2024 ; Huang et al. 2024a ) . Recently, researchers have begun to explore using such interaction for various data generation, including instructions ( Mitra et al. 2024 ) , reasoning chains ( Yang et al. 2024b ) , and environment-aware trajectories ( Sun et al. 2024b ; Hu et al. 2025 ) . Leveraging interaction for flexible and scalable data construction is emerging as a promising direction ( Luo et al. 2024a ; Khan et al. 2025 ) . This work takes an initial step toward applying interaction to instruction-code synthesis. Code - centric Data Synthesis. The synthesis of instruction-code data traces back to early symbolic augmentation methods ( Wang et al. 2017 ; Feng et al. 2018 ) , which augment existing code using program transformations. Recent efforts move beyond static heuristics and leverage LLMs to generate instruction-code pairs at scale with prompting ( Chaudhary 2023 ; Wang et al. 2023 ; Xu et al. 2025b ) , as well as through human-in-the-loop interactions ( Wei et al. 2024a ) . Typically,

Read the complete text