Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
Jingcheng Hu 1,2∗ , Yinmin Zhang 1 , Qi Han 1 , Daxin Jiang 1 , Xiangyu Zhang 1 , Heung-Yeung Shum 2 1 StepFun, 2 Tsinghua University GitHub: https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero , HuggingFace: https://huggingface.co/Open-Reasoner-Zero . · 2025
Read Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model by Jingcheng Hu 1,2∗ , Yinmin Zhang 1 , Qi Han 1 , Daxin Jiang 1 , Xiangyu Zhang 1 , Heung-Yeung Shum 2 1 StepFun, 2 Tsinghua University GitHub: https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero , HuggingFace: https://huggingface.co/Open-Reasoner-Zero . (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24290.txt.
Introduction
Jingcheng Hu and five co-authors released in 2025 an open recipe for training a base language model to reason, using reinforcement learning at scale. Their system keeps the method plain: a standard learning rule with simple rule-based rewards and no penalty for drifting from the base model. Even so, skill and answer length scaled up together.
On the same base model used by a famous closed pipeline, their version scored higher on three hard test sets while using a tenth of the training steps. The paper also maps training dynamics and design choices. Open detail lets others copy the feat.
Catalog record
| CREATOR | Jingcheng Hu 1,2∗ , Yinmin Zhang 1 , Qi Han 1 , Daxin Jiang 1 , Xiangyu Zhang 1 , Heung-Yeung Shum 2 1 StepFun, 2 Tsinghua University GitHub: https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero , HuggingFace: https://huggingface.co/Open-Reasoner-Zero . |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-24290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model Jingcheng Hu 1,2∗ , Yinmin Zhang 1 , Qi Han 1 , Daxin Jiang 1 , Xiangyu Zhang 1 , Heung-Yeung Shum 2 1 StepFun, 2 Tsinghua University GitHub: https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero , HuggingFace: https://huggingface.co/Open-Reasoner-Zero . Abstract We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE ( λ = 1 \lambda=1 , γ = 1 \gamma=1 ) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both response length and benchmark performance, similar to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating remarkable efficiency—requiring only a tenth of the training steps, compared to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our source code, parameter settings, training data, and model weights across various sizes. Figure 1: Evaluation performance of Open-Reasoner-Zero-{7B, 32B} on benchmarks (averaged on 16 responses) during training. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, Open-Reasoner-Zero-32B achieves superior performance on AIME2024, MATH500, and GPQA Diamond benchmark-requiring only a tenth of the training steps. 1 Introduction Large-scale reinforcement learning (RL) training of language models on reasoning tasks has emerged as a promising paradigm for mastering complex problem-solving skills. Recent breakthroughs, particularly OpenAI’s o1 [ 1 ] and DeepSeek’s R1-Zero [ 2 ] , have demonstrated remarkable training time scaling phenomenon: as the training computation scales up, both the model’s benchmark performance and response length consistently and steadily increase without any sign of saturation. Inspired by these advancements, we aim to explore this new scaling phenomenon by conducting large-scale RL training, even applying it directly to base models, an approach we refer to as Reasoner-Zero training. In this work, we introduce Open-Reasoner-Zero (ORZ), the first open-source implementation of large-scale reasoning-oriented RL training on large language models (LLMs) with our best practices, designed to be robust, scalable and simple-to-follow. Under Reasoner-Zero paradigm, LLMs are trained to master diverse reasoning skills under verifiable rewards, spanning arithmetic, logic, coding and common-sense reasoning ( e.g. , scientific problems, numerical reasoning, natural language understanding and even creative writing). While DeepSeek’s R1-Zero outlined their training pipeline briefly, we provide a comprehensive study of our training strategy, with in-depth insights into overcoming common challenges such as training instability, stagnating response length, benchmark performance plateaus, and reward design. Our goal is to democratize advanced RL training techniques accessible to the broader research community. Our proposed Open-Reasoner-Zero outperforms the DeepSeek-R1-Zero, with the same Qwen-32B base, on AIME24, MATH500, and GPQA Diamond, yet only 1/10 the training steps. Through extensive ablation studies, we summarize some key findings and lessons learned from our exploration. Specifically, vanilla PPO using GAE ( \lambda=1 and \gamma=1 ) and without any KL-related regularization, combined with a straightforward rule-based reward, is sufficient to achieve steady scalability in both response length and benchmark performances across varying model sizes and training data scales. ORZ’s stable scaling resonates well with the bitter lesson [ 3 ] : the most significant performance improvements stem from the scale of training data, model size, and training iterations, rather than the complexity of design choices. The most critical thing is how to design a simple and effective RL algorithm to scale up the training process. We release all of our training resources, including code, parameters, data, and model weights. We are excited to share this breakthrough of scale-up RL, along with the lessons we learned with the research community. This sharing empowers researchers to go beyond merely using the prepared results ( e.g. , APIs or model weights) and instead become active participants in this transformative moment in AI development themselves. Additionally, we provide recommended practices for 0.5B and 1.5B model variants to assist a wider range of researchers. Our primary contributions are as follow: 1. We provide a fully open-source implementation of large-scale RL training directly on a base LLM, a strategy we refer to as Open-Reasoner-Zero. 2. We share empirical insights and lessons we learned from frustrating failures and exciting breakthroughs during our scaling-up journey. 3. We release a comprehensive training resources including code, parameter settings, data, and model weights to the research community. Our findings indicate that model performance continues to improve with more training data, showing no signs of saturation. In light of this, we are releasing our training datasets and calling on the community to contribute additional data to collectively advance the frontiers of LLMs. Figure 2: Train-time Scale up on Train Reward and Response Length of Open-Reasoner-Zero (ORZ) - {0.5B, 1.5B, 7B, 32B}. Train Reward and Response Length increase steadily, demonstrating consistent scalability across model sizes. Interestingly, the ORZ-32B Response Length exhibits fluctuations without negatively impacting training stability, highlighting the robustness of our minimalist recipe. 2 Scale-up Reinforcement Learning from a Base Model In this section, we describe the strategy and critical components for scale-up reasoning-oriented reinforcement learning (RL) directly from a base model. First, we introduce the basic yet critical settings for our scale-up RL training from a base model, including data curation, reward function, and detailed settings of the Proximal Policy Optimization (PPO) [ 4 ] algorithm. We then discuss key insights derived from our comprehensive ablation experiments that enable successful scale-up RL training. 2.1 Basic Settings We conduct our experiments utilizing the Qwen2.5-{7B, 32B} as our base model [ 5 ] , and directly starting the large-scale RL training without any fine-tuning ( e.g. , distillation or SFT) [ 6 , 7 ] . Building upon the Qwen2.5-{7B, 32B} base model, we scale up the standard PPO algorithm [ 4 ] for reasoning-oriented RL training, with careful consideration of scalability and robustness. Our training data comprises tens of thousands of carefully curated question and answer pairs consisting of Math and Reasoning tasks, designed specifically for enhancing models’ capability in diverse and complex problem-solving scenarios. Inspired by DeepSeek-R1 [ 2 ] , we design our prompt template to elicit the model to utilize inference computation, gradually mastering the reasoning ability for complex tasks, as shown in Table 1 . Furthermore, we develop an efficient and easy-to-use large-scale RL training framework based on OpenRLHF [ 8 ] , by introducing a more flexible trainer, enabling GPU collocation generation, and training with offload and backload support. In the following sections, we provide detailed settings for our scale-up RL training from a base model. A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. User: You must put your answer inside <answer> </answer> tags, i.e., <answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed{} tag. {{prompt}} Assistant: <think> Table 1: Template for Open-Reasoner-Zero. prompt will be replaced with the specific reasoning question during training. 2.1.1 Dataset In this section, we introduce our carefully