o1-Coder: an o1 Replication for Coding
Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong & Jitao Sang · 2024
Read o1-Coder: an o1 Replication for Coding by Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong & Jitao Sang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00154.txt.
Introduction
Yuxiang Zhang and six co-authors present a technical report in 2024 on O1-CODER, their attempt to copy OpenAI's o1 model for coding tasks. The o1 model is known for slow, step by step reasoning. The team wanted the same style of thinking in a model that writes code. Their design joins reinforcement learning with Monte Carlo Tree Search, a method that explores many paths before it picks one.
A trained test case generator supplies standard checks for the code the model writes. Tree search then builds training data that keeps the reasoning steps, not just the final code. The policy model is fine tuned in rounds, first to draft pseudocode and then to write full code. The report also weighs the chances and hurdles of using such models in real work. All code, data sets, and models are released in public.
Catalog record
| CREATOR | Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong & Jitao Sang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
o1-Coder: an o1 Replication for Coding Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong & Jitao Sang Abstract The technical report introduces O1-CODER, an attempt to replicate OpenAI’s o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model’s System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode and then generate the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for world model construction. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models are disclosed at https://github.com/ADaM-BJTU/O1-CODER . 1 Introduction OpenAI recently introduced the o1 model ( OpenAI 2024 ) , which has demonstrated impressive system-2 thinking capabilities. This model represents a significant advancement in AI’s ability to perform complex reasoning tasks that require higher-order cognitive functions. Following its release, numerous analysis and replication efforts have emerged, demonstrating the growing interest in reasoning models. Notable works include g1 ( Benjamin Klieger 2024 ) , OpenO1 ( ope 2024 ) , O1-Journey ( GAIR-NLP 2024 ) , OpenR ( Team 2024 ) , LLaMA-O1 ( SimpleBerry 2024 ) , LLaMA-Berry ( Zhang et al. 2024 ) , Steiner ( Ji 2024 ) , Thinking Claude ( Richards Tu 2024 ) , LLaVA-o1 ( Xu et al. 2024 ) , and several industrial releases such as k0-math, DeepSeek-R1-Lite, Macro-o1 ( Zhao et al. 2024 ) , Skywork o1, QwQ ( Qwen Team 2024 ) , and InternThinker ( Shanghai AI Lab 2024 ) (illustrated in Fig. 1 ). Prior to the o1 model, large language models (LLMs) primarily exhibited System-1 capabilities, characterized by fast, intuitive responses. These models were trained on datasets consisting mainly of question-answer (Q,A) pairs, lacking the intermediate reasoning steps that involve deliberate and analytical processing. This stems from the fact that humans rarely record their thought processes on the internet or elsewhere. Traditionally, techniques such as Chain-of-Thought (CoT) prompting were used to guide models in generating step-by-step reasoning before arriving at an answer. A more direct and effective way is to create datasets including the reasoning sequences, e.g., (Q,...,S_{i},...,A) , where S_{i} represents an individual reasoning step leading to the final answer. Thus, a second approach to enhancing System-2 reasoning capabilities is supervised fine-tuning (SFT). However, most publicly available data is recorded in a question-answer form, and annotating or distilling such data, especially for complex tasks, is both costly and challenging. In this study, we aim to explore in the absence of reasoning process data, and thus opt for the third approach of reinforcement learning (RL). It is widely believed that o1 addresses the lack of reasoning data by combining reinforcement learning with pretraining. Reinforcement learning is well known for its ability to explore and discover new strategies rather than relying on predefined data in the past decade. Looking back at key developments in machine learning, we can see that deep learning and large-scale pretraining have driven transformations in model architecture and the requirements for labeled data, respectively. In contrast, reinforcement learning addresses a different aspect of transformation on the objective function. In situations where explicit guidance or clear goals are absent, RL exploits exploration to search for new knowledge and solutions. Therefore, combining pretraining with RL creates a powerful synergy of learning and search, where pretraining compresses existing human knowledge, and RL enables the model to explore new possibilities. We chose coding tasks to explore how to employ RL to generate and refine reasoning data. Coding is a typical task that requires System-2 thinking, involving careful, logical, and step-by-step problem-solving. Moreover, coding can serve as a foundational skill for solving many other complex problems. This report presents our attempt to replicate o1 with a specific focus on coding tasks. The approach integrates RL and Monte Carlo Tree Search (MCTS) to enable self-play, allowing the model to continually generate reasoning data and enhance its System-2 capabilities. Figure 1: o1 replication efforts: upper part from academic institutions and open-source communities, and lower part from the industry. 2 Framework Overview There are two main challenges to address for applying self-play RL to code generation. The first challenge is result evaluation, i.e., assessing the quality of the generated code. Unlike tasks such as Go or mathematics, where results can be directly evaluated based on game rules or correct answers, evaluating code requires running the generated code within a testing environment and verifying it against test cases. We cannot assume that code datasets will always provide sufficient test cases. The second challenge involves defining the thinking and search behaviors, i.e., determining the state transition and the granularity of process rewards. For code generation, the key question is how to design the reasoning process and the space of policies to guide the model’s behavior effectively. To address the first challenge, we propose training a Test Case Generator (TCG), which automatically generates test cases based on the question and the ground-truth code 1 1 1 We also propose an alternative approach where test cases are generated based solely on the question. In addition to be capable of utilizing code datasets that only provide questions, it can also be applied during the inference phase, enabling online reasoning without the need for ground-truth code. . This approach will help build a standardized code testing environment, providing result rewards for reinforcement learning. For the second challenge, two possible approaches can be considered. One is “think before acting”, where the model first forms a complete chain-of-thought and then generates the final answer all at once. The other approach, “think while acting” ( Zelikman et al. 2024 ) , involves generating parts of the answer while simultaneously reasoning through the task. We chose the former approach in this study. For code generation, this means first thinking through and writing out a detailed pseudocode, which is then used to generate the final executable code. The advantages are two-fold: adaptability, as the same pseudocode can lead to different concrete code implementations; and controllable granularity, as adjusting the level of detail in the pseudocode can control the granularity of the reasoning/search behavior. The outlined framework is provided in Algorithm 1 , which consists of six steps. (1) The first step is training the test case generator (TCG) \gamma_{\text{TCG}} , which is responsible for automatically generating test cases based on the question. (2) In the second step, we run MCTS on the original code dataset to generate code dataset with reasoning processes \mathcal{D}_{\text{process}} , including a validity indicator to distinguish between correct and incorrect reasoning steps. (3) Once we have data that includes the reasoning process, the third step is to fine-tune the policy model \pi_{\theta} , training it to behave in a “think before acting” manner. (4) The reasoning process data can also be used to initialize the process reward model (PRM) \rho_{\text{PRM}} , which evaluates the quality of reasoning steps. (5) The fifth step is the