rStar2-Agent: Agentic Reasoning Technical Report
Ning Shang Yifei Liu Yi Zhu Li Lyna Zhang Weijiang Xu Xinyu Guan Buze Zhang Bingcheng Dong Xudong Zhou Bowen Zhang, Ying Xin Ziming Miao Scarlett Li Fan Yang Mao Yang Microsoft Research · 2025
Read rStar2-Agent: Agentic Reasoning Technical Report by Ning Shang Yifei Liu Yi Zhu Li Lyna Zhang Weijiang Xu Xinyu Guan Buze Zhang Bingcheng Dong Xudong Zhou Bowen Zhang, Ying Xin Ziming Miao Scarlett Li Fan Yang Mao Yang Microsoft Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20722.txt.
Introduction
Ning Shang and a large team of co-authors at Microsoft Research present a 2025 technical report on a math reasoning model, rStar2-Agent. The model has fourteen billion parameters and learns by doing: it pauses to think before calling a code tool, reads the output, and reworks its steps. That loop lets it explore and check hard problems on its own.
Three supports make the training work. A fast, stable code runner keeps the trials cheap. A learning rule resamples good runs to fight noise from the tools. And a staged recipe starts with plain tuning before the hard stages. In only 510 training steps over one week, the model reached top scores on two math contest sets. It beat a far larger rival while giving shorter answers. Skills also carried to science and tool-use tasks.
Catalog record
| CREATOR | Ning Shang Yifei Liu Yi Zhu Li Lyna Zhang Weijiang Xu Xinyu Guan Buze Zhang Bingcheng Dong Xudong Zhou Bowen Zhang, Ying Xin Ziming Miao Scarlett Li Fan Yang Mao Yang Microsoft Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
rStar2-Agent: Agentic Reasoning Technical Report Ning Shang Yifei Liu Yi Zhu Li Lyna Zhang Weijiang Xu Xinyu Guan Buze Zhang Bingcheng Dong Xudong Zhou Bowen Zhang, Ying Xin Ziming Miao Scarlett Li Fan Yang Mao Yang Microsoft Research Abstract We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance. Beyond current long CoT, the model demonstrates advanced cognitive behaviors, such as thinking carefully before using Python coding tools and reflecting on code execution feedback to autonomously explore, verify, and refine intermediate steps in complex problem-solving. This capability is enabled through three key innovations that makes agentic RL effective at scale: (i) an efficient RL infrastructure with a reliable Python code environment that supports high-throughput execution and mitigates the high rollout costs, enabling training on limited GPU resources (64 MI300X GPUs); (ii) GRPO-RoC , an agentic RL algorithm with a Resample-on-Correct rollout strategy that addresses the inherent environment noises from coding tools, allowing the model to reason more effectively in a code environment; (iii) An efficient agent training recipe that starts with non-reasoning SFT and progresses through multi-RL stages, yielding advanced cognitive abilities with minimal compute cost. To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 (671B) with significantly shorter responses. Beyond mathematics, rStar2-Agent-14B also demonstrates strong generalization to alignment, scientific reasoning, and agentic tool-use tasks. Code and training recipes are available at https://github.com/microsoft/rStar . Model AIME24 AIME25 HMMT25 OpenAI o3-mini (medium) 79.6 77.0 53.0 DeepSeek-R1 (671B) 79.8 70.0 44.4 DeepSeek-R1-Zero (671B) 71.0 53.3 46.0 Claude-Opus-4.0 (Think) 76.0 69.2 - QWQ-32B 79.5 65.8 47.5 rStar2-Agent-14B 80.6 69.8 52.7 Figure 1: rStar2-Agent-14B reaches frontier-level math reasoning in just 510 RL training steps. $*$ $*$ footnotetext: The first four authors contributed equally $\dagger$ $\dagger$ footnotetext: Project leaders; correspondence to [email protected] and [email protected] 1 Introduction Test-time scaling has recently driven substantial advances in complex reasoning ( Guan et al. 2025 ; Team et al. 2025 ) . Leading models such as OpenAI o-series ( Jaech et al. 2024 ; OpenAI 2024 ) , DeepSeek-R1 ( Guo et al. 2025 ) and Gemini-2.5 ( DeepMind 2025 ) show that extending the Chain-of-Thought (CoT), in essence “ thinking longer ”, can markedly improve performance, especially when optimized through large-scale reinforcement learning with verifiable rewards (RLVR). However, long CoT remains fundamentally limited for hard problems prone to subtle intermediate errors or requiring creative shifts in reasoning. In these cases, models depend on internal self-reflection, which often fails to detect mistakes ( Sui et al. 2025 ) or to self-correct when the initial approach is flawed. To move beyond merely “thinking longer”, we aim to enable models to “ think smarter ” by developing more advanced cognitive abilities that autonomously utilize the right tools to reason, validate, and learn from the feedback signals provided by the tool environment. We incentivize these abilities through agentic reinforcement learning , where the model interacts with tools insides the dedicated tool environment and adapts its reasoning based on the feedback it receives. Crucially, not all tools or environments are equally effective; a valuable environment must be deployable and provide accurate, verifiable signals that guide the model toward stronger reasoning paths. In this work, we focus on Python coding tools and the interpreter as the environment for agentic reinforcement learning. Python coding tools broaden the model’s action space, enabling exploration of alternative solutions and verification of intermediate steps, thereby complementing internal self-reflection when long CoT alone is insufficient. However, effectively scaling agentic reinforcement learning poses significant challenges. First, the inherent complexity of coding tools and Python interpreter introduces environment noise into the reasoning process. When the model inevitably generates syntactically or logically incorrect code, the resulting environment feedback (e.g., error message) can cause it to waste valuable tokens correcting mistakes rather than advancing reasoning. Unfortunately, current RL methods ( Shao et al. 2024 ; Guo et al. 2025 ) , which rely primarily on outcome-only rewards, exacerbates this issue because trajectories with failed intermediate tool calls still receive positive reward if the final answer is correct. As a result, the model treats errors as acceptable and produces lengthy, low-quality reasoning trajectory. Second, large-scale agentic RL training imposes substantial infrastructure demands. A single training batch can trigger tens of thousands of concurrent tool calls, making it challenging to construct a reliable and responsive code execution environment. Moreover, agentic rollouts with environment interactions amplify the rollout inefficiencies in standard RL systems, significantly slowing the overall training process. In this work, we introduce rStar2-Agent , a novel agentic reinforcement learning approach that trains a 14B reasoning model, rStar2-Agent-14B, to reach frontier-level performance, rivaling or surpassing the 671B DeepSeek-R1. rStar2-Agent incorporates three key innovations. First, we build an efficient and reliable infrastructure for large-scale agentic RL. We construct a high-throughput, isolated code environment capable of handling 45K concurrent tool calls, with execution feedback returned in just 0.3 seconds on average. To address RL rollout inefficiencies, we introduce a load-balanced rollout scheduler that dynamically allocates rollout requests based on available KV cache capacity across GPUs to maximize computational utilization. This infrastructure enables efficient RL training even with limited GPU resources. Using 64 \times MI300X GPUs, we complete rStar2-Agent-14B training in just one week. Second, to enable effective agentic reinforcement learning in a code environment, we propose Group Relative Policy Optimization with Resampling on Correct (GRPO-RoC) , which integrates GRPO with a Resample-On-Correct (RoC) rollout strategy to address environment-induced noise under sparse, outcome-only rewards. Specifically, RoC first oversamples a larger group of rollouts and then downsamples to the standard batch size. Positive trajectories are filtered to retain only the highest-quality ones with minimal tool-induced errors or formatting issues, while negative trajectories are uniformly downsampled. This simple yet effective asymmetric sampling preserves diverse failure modes as informative negative signals while emphasizing higher-quality success cases for positive supervision. Compared to methods that explicitly penalize tool-use errors in the reward function ( Qian et al. 2025 ; Li et al. 2025 ; Kimi ) , GRPO-RoC improves training stability and avoids reward-hacking risks. By learning from cleaner, higher-quality positive trajectories, the model not only improves Python coding tool usage but also exhibits advanced cognitive abilities, reasoning more effectively and concisely under realistic code-environment interactions. Finally, we present our training recipe that boosts a 14B pre-trained base model to frontier-level math reasoning with minimal compute. Unlike prior works that apply reasoning-heavy SFT before RL ( Liu et al. 2025a ; Feng et al. 2025 ; Team 2025 ; Seed et al. 2025 ) , we begin with a non-reasoning SFT stage solely to instill general instruction-following, coding tool usage, and formatting, without enhancing reasoning. This avoids potential SFT overfitting and keeps initial average responses short, allowing RL to more effectively cultivate reasoning while fully exploiting the model’s pre-trained capability. We then conduct multi-stage RL training with GRPO-RoC, gradually increasing task difficulty and maximum training length. Unlike prior RL methods that heavily scale rollouts to 16K \rightarrow 48K or more ( Chen et al. 2025 ; Xiaomi et al. 2025 ) , we limit each stage to shorter lengths (8K