Zero Reinforcement Learning Towards General Domains

Yuyuan Zeng, Yufei Huang, Can Xu, Qingfeng Sun, Jianfeng Yan, Guanghui Xu, Tao Yang, Fengzong Lian, LLM Department, Tencent · 2025

Read Zero Reinforcement Learning Towards General Domains by Yuyuan Zeng, Yufei Huang, Can Xu, Qingfeng Sun, Jianfeng Yan, Guanghui Xu, Tao Yang, Fengzong Lian, LLM Department, Tencent (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25528.txt.

Introduction

Yuyuan Zeng led seven co-authors on zero reinforcement learning. Yufei Huang, Can Xu, and Qingfeng Sun helped. Jianfeng Yan, Guanghui Xu, and Tao Yang joined. Fengzong Lian closes the eight-author team. Their 2025 paper improves thinking across general domains. Zero reinforcement learning trains pretrained models directly with rewards.

No supervised fine-tuning phase is needed beforehand. Current research focuses on easily verifiable domains mainly. Math and programming offer clear reward signals there. Diverse scenarios lack straightforward verification, though. Their paradigm mixes verifiable rewards with generative rewards. A generative reward model judges non-verifiable tasks. Multi-task training transfers thinking between domain types. A smooth length penalty mitigates reward hacking risks. It encourages more full thinking tokens in general domains. Tests used Qwen3 base models at two sizes. Superior thinking results appeared on extensive thinking tasks. General tasks also better under their method.

Catalog record

CREATORYuyuan Zeng, Yufei Huang, Can Xu, Qingfeng Sun, Jianfeng Yan, Guanghui Xu, Tao Yang, Fengzong Lian, LLM Department, Tencent
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25528

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero Reinforcement Learning Towards General Domains Yuyuan Zeng, Yufei Huang, Can Xu, Qingfeng Sun, Jianfeng Yan, Guanghui Xu, Tao Yang, Fengzong Lian, LLM Department, Tencent Abstract Zero Reinforcement Learning (Zero-RL) has proven to be an effective approach for enhancing the reasoning capabilities of large language models (LLMs) by directly applying reinforcement learning with verifiable rewards on pretrained models, without the need for a supervised fine-tuning phase. However, current research on zero-RL primarily focuses on domains with easily verifiable reward signals, such as mathematics, programming, and other reasoning tasks. The challenge of eliciting reasoning abilities in more diverse scenarios, where verification is not straightforward, remains underexplored. To address this gap, we propose a novel zero-RL paradigm designed to improve a model’s reasoning ability across both verifiable and non-verifiable domains. By combining verifiable rewards with a generative reward model, we conduct multi-task zero-RL training across both domains, facilitating the transfer of reasoning capabilities between them. Furthermore, to mitigate reward hacking in the generative reward model, we design a smooth length penalty that encourages the generation of more comprehensive thinking tokens in general domains. Experimental results on Qwen3-8B-Base and Qwen3-14B-Base demonstrate that our approach achieves superior reasoning performance, not only on tasks requiring extensive reasoning but also on more general tasks. 1 Introduction Recent advances in large language models (LLMs) have unlocked significant potential for artificial intelligence across diverse domains. In particular, efforts such as OpenAI-o1 ( OpenAI 2024 ) and DeepSeek-R1 ( Guo et al. 2025b ) have introduced long chain-of-thought (CoT) reasoning, which represents a breakthrough in enhancing the reasoning capabilities of LLMs. A central technique in this progress is zero reinforcement learning proposed in DeepSeek-R1-Zero ( Guo et al. 2025b ) , which converts large language models into large reasoning models (LRMs). Specifically, zero-RL directly trains pretrained LLMs using reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) and Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) with verifiable reward signals, and has demonstrated remarkable improvements on verifiable domains like mathematics and programming ( Yu et al. 2025 ; Zeng et al. 2025b ) . There has been a surge of interest in expanding zero-RL methods to more diverse reasoning tasks, as exemplified by General-Reasoner ( Ma et al. 2025 ) and Nemotron-Crossthink ( Akter et al. 2025 ) . However, existing work on general zero-RL-based reasoning still largely focuses on STEM domains, where problems have well-defined ground truths that allow straightforward verification of model outputs. In contrast, applying zero-RL to open-ended tasks remains underexplored. The key challenge lies in the fact that responses in open-ended tasks are difficult to verify, making it hard to obtain reliable and specific reward signals. To address these challenges, we propose a unified zero-RL framework (General Zero-RL) which integrates both verifiable rewards and generative reward models. Unlike conventional zero-RL training on reasoning data, our preliminary experiments reveal that training solely on general-domain data struggles to elicit meaningful reasoning behaviors (e.g., step-by-step analysis, self-reflection, and backtracking). In such cases, the model often tends to output final answers without a substantive thinking process, or with only superficial reasoning steps. To mitigate this, we adopt a multi-task zero-RL training strategy that transfers reasoning behaviors across diverse domains by jointly training on both general and reasoning data. Specifically, for reasoning tasks with ground-truth answers (treated as verifiable tasks), we use the final accuracy assessed by a verifier model as the reward signal. For general tasks with open-ended answers, we employ a generative reward model that evaluates the quality of responses and provides corresponding reward scores. However, since model-based reward models are often criticized for favoring longer responses ( Saito et al. 2023 ; Hu et al. 2024 ) , we observe that on general data the model tends to produce unnecessarily lengthened answers rather than substantive reasoning processes. To mitigate this reward hacking issue, we introduce a smooth length penalty, which penalizes the length difference between the length of the reasoning process and the answer content to prevent the model from producing verbose response. This length penalty regularization yields a more stable increase in response length during RL training. Furthermore, by gradually expanding the maximum allowable response length during training ( Wang et al. 2025 ) , we can avoid sudden spikes in response length and further stabilize model optimization. To assess the effectiveness of our approach, we conduct extensive experiments on Qwen3-8B-Base and Qwen3-14B-Base ( Yang et al. 2025 ) , and evaluate our models across three categories of tasks: Math Reasoning , including MATH-500 ( Lightman et al. 2023 ) , AIME ( AIME 2025 ) , and OlympiadBench ( He et al. 2024 ) ; General Reasoning , including MMLU-Pro ( Wang et al. 2024 ) , GPQA-Diamond ( Rein et al. 2024 ) , SuperGPQA ( Du et al. 2025 ) and BBEH ( Kazemi et al. 2025 ) ; and General Tasks , such as Arena-Hard ( Li et al. 2024 ) , WritingBench ( Wu et al. 2025 ) , WildBench v2 ( Lin et al. 2024 ) and AlpacaEval2.0 ( Dubois et al. 2024 ) . On reasoning tasks, our General Zero-RL models not only outperform other zero-RL trained models of comparable size, but also surpass DeepSeek-R1-Zero-Qwen-32B ( Guo et al. 2025b ) by a significant margin. On general tasks, which have received little attention in prior zero-RL research, our models also achieve competitive results and generate coherent, meaningful reasoning content. Ablation studies further confirm the crucial role of our multi-task learning and length penalty strategy. Collectively, these findings demonstrate that our multi-task zero-RL training effectively enhances the reasoning capabilities of pretrained LLMs across both verifiable and non-verifiable tasks. Overall, our contributions in this paper can be summarized as follows: • We propose a unified zero reinforcement learning (zero-RL) framework that integrates both verifiable and non-verifiable tasks, enabling the elicitation of reasoning capabilities across a broad range of domains, extending beyond traditional reasoning tasks. • We introduce a novel length penalty tailored for general-domain data, which mitigates reward hacking problems and stabilizes the zero-RL training process. • We demonstrate that reasoning behaviors acquired through our multi-task zero-RL paradigm can generalize effectively across diverse task domains. 2 Related Works 2.1 Zero Reinforcement Learning in LLMs Large reasoning models have significantly improved the capabilities of large language models in solving complex problems. Following this direction, representative works such as DeepSeek-R1 ( Guo et al. 2025b ) and OpenAI-O1 ( OpenAI 2024 ) have demonstrated strong capabilities in mathematics and programming. In particular, DeepSeek-R1-Zero ( Guo et al. 2025b ) demonstrated that directly applying reinforcement learning on the base model can effectively uncover significant reasoning capabilities without supervised fine-tuning. This zero reinforcement learning paradigm has achieved a great success in the domain of Reinforcement Learning with Verifiable Rewards (RLVR) ( Zeng et al. 2025b ; Yu et al. 2025 ) . Recently, several works tend to explore zero-RL in diverse general domains, such as General-Reasoner ( Ma et al. 2025 ) , Nemotron-Crossthink ( Akter et al. 2025 ) and RLMT ( Bhaskar et al. 2025 ) . Both General-Reasoner ( Ma et al. 2025 ) and Nemotron-Crossthink ( Akter et al. 2025 ) incorporated

Read the complete text