Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang Dongchun Xie, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), University of California, Merced Sun Yat-sen University MBZUAI [email protected] Project Repo : https://github.com/yangzhch6/DARS · 2025

Read Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration by Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang Dongchun Xie, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), University of California, Merced Sun Yat-sen University MBZUAI [email protected] Project Repo : https://github.com/yangzhch6/DARS (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13755.txt.

Introduction

Zhicheng Yang and co-authors study how to train reasoning models in a 2025 paper. The training method rewards answers that can be checked as correct, but it tends to down-weight the hardest problems. Those are exactly the problems that could teach the most, so learning stalls.

Their method, DARS, gives hard problems extra attempts in stages, rebalancing the training mix toward depth. Extra attempts alone did not help and could hurt, the team found. Widening the batch of training cases, the breadth side, did help, by keeping the model's choices more varied. A combined version improved both single-answer accuracy and the chance of success over many attempts. The authors frame depth and breadth as separate levers that work best together.

Catalog record

CREATORZhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang Dongchun Xie, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), University of California, Merced Sun Yat-sen University MBZUAI [email protected] Project Repo : https://github.com/yangzhch6/DARS
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang Dongchun Xie, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), University of California, Merced Sun Yat-sen University MBZUAI [email protected] Project Repo : https://github.com/yangzhch6/DARS Abstract Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth —the hardest problem a model can sample; Breadth —the number of instances consumed in a single iteration. We dissect the popular GRPO algorithm and reveal a systematic bias: the cumulative-advantage disproportionately weights samples with medium accuracy, while down-weighting the low-accuracy instances that are crucial for pushing reasoning boundaries. To rectify the depth neglect, we introduce Difficulty Adaptive Rollout Sampling ( DARS ), which re-weights hard problems through targeted multi-stage rollouts, thereby increasing the number of positive rollouts for hard problems. Empirically, naively enlarging rollout size only accelerates convergence and even hurts Pass@K . Our DARS, in contrast, delivers consistent Pass@K gains without extra inference cost at convergence. Just as we adaptively expanded the depth of exploration, we now ask whether aggressively scaling the breadth of training data can further amplify reasoning gains. To this end, we intensely scale batch size and replace PPO’s mini-batch iterations with full-batch updates over multiple epochs. Increasing breadth significantly enhances Pass@1 performance. Large-breadth training sustains high token-level entropy, indicating continued exploration and reduced gradient noise. We further present DARS-Breadth, which augments DARS with large breadth, and demonstrate simultaneous gains in Pass@K and Pass@1 . The results confirm that breadth and adaptive exploration across depth operate as orthogonal dimensions in RLVR, which are key to unleashing the reasoning power of RLVR. Figure 1: Training dynamics of Pass@1 and Pass@K performance. We show that our DARS significantly improves Pass@K performance and is complementary to large breadth scaling to further improve Pass@1 performance. 1 Introduction The emergence of reasoning-centric Large Language Models (LLMs) exemplified by OpenAI-o1 ( Jaech et al. 2024 ) , DeepSeek-R1 ( Guo et al. 2025 ) , and Kimi-1.5 ( Team et al. 2025 ) , has pushed the frontier of LLM capability, especially for demanding tasks in complex reasoning such as mathematics and programming. Unlike conventional instruction tuning that relies on human-labeled data or RLHF pipelines that demand an auxiliary, well-trained reward model ( Ouyang et al. 2022a ; Achiam et al. 2023 ; Grattafiori et al. 2024 ) , this leap is driven by large-scale Reinforcement Learning with Verifiable Rewards (RLVR; Guo et al. 2025 ; Zeng et al. 2025 ) for which correctness can be automatically and deterministically checked. The rewards of RLVR are granted solely when a model’s output matches the ground-truth mathematical answer or passes all unit tests for code, allowing scalable verification without manual labeling. RLVR is now regarded as a promising path toward self-evolving LLMs, potentially bringing us closer to more powerful intelligence. However, existing RLVR frameworks inadequately address the interplay between exploration depth (difficulty scaling) and breadth (iteration instance quantity scaling), which leads to insufficient performance gain for both Pass@1 and Pass@K . In this paper, we conduct a systematic analysis of two under-exploited dimensions in RLVR: Depth and Breadth. For the dimension of depth , our investigation reveals that existing methods of GRPO ( Shao et al. 2024 ) and its variants ( Yu et al. 2025 ; Liu et al. 2025b ) , while adept at estimating the advantage of a single rollout, are undermined by a distorted cumulative advantage at the group level. This distortion disproportionately allocates attention to instances of medium difficulty, neglecting high-difficulty instances indispensable for complex reasoning, as illustrated in Figure 3 . This bias fundamentally limits depth, the hardest problems a model can learn to solve, and constrains Pass@K performance. To counteract this depth neglect, we propose Difficulty-Adaptive Rollout Sampling ( DARS ). DARS performs a lightweight first-stage rollout to estimate per-problem accuracies, then allocates additional compute via targeted multi-stage rollouts to low-accuracy problems. By expanded sampling on hard problems, DARS re-weights the cumulative advantage, making it easier for LLMs to learn ‘deep’ samples and improving Pass@K performance. We further identify breadth as the instance quantity consumed in a single iteration. We observe that breadth has a significant impact on the LLM’s performance and continuous exploration capability, as shown in Figure 5 . We significantly increase the training batch size and replace PPO-minibatch updates with full-batch updates for multiple PPO-epochs. This seemingly simple change dramatically improves Pass@1 and sustains high token-level entropy throughout training, suggesting that breadth acts as implicit entropy regularization that delays premature convergence. Importantly, the gains from breadth are complementary to those from depth: we present DARS-Breadth that combines our DARS with large-breadth training, producing simultaneous boosts in both Pass@K and Pass@1 . Our contributions can be summarized as follows: • We conduct a systematic analysis on depth and breadth in RLVR, and uncover the depth bias in GRPO: cumulative advantage silently underweights low-accuracy, high-difficulty samples, capping Pass@K performance. • We introduce DARS, which reallocates compute from medium difficulty problems to the hardest problems via multi-stage rollout sampling. DARS re-weights the cumulative advantage distribution and quantitatively expands the sparse reward signals for difficult problems. In practice, DARS significantly improves Pass@K over multiple benchmarks. • We further illustrate that large breadth in RLVR training matters for the Pass@1 performance. Moreover, by combining DARS with large breadth training, we reveal the complementarity of Depth and Breadth in RLVR and acquire simultaneous boosts in both Pass@K and Pass@1 performance. 2 Understanding RLVR from Depth and Breadth In this section, we identify the two dimensions of Depth and Breadth and examine their impact on the Pass@1 and Pass@K metrics during the RLVR process. For training details and specific experimental settings, please refer to Section 4.1 . 2.1 Depth: The Hardest Problem Sampled in RLVR We first identify Depth as the hardest problem that can be correctly answered in the RLVR training process. In the GRPO training process, groups whose entire rollouts yield incorrect answers suffer from gradient vanishing. Hence, sampling high-difficulty questions with correct reasoning paths is crucial for LLM training. In this subsection, we first show that merely increasing rollout size does not consistently yield significant gains in Pass@K performance, and sometimes can even be harmful. We then quantify GRPO’s cumulative advantage and highlight its under-weighting of high-difficulty samples. Figure 2: Training dynamics of Pass@1 and Pass@K performance of Qwen2.5-Math-1.5b and Qwen2.5-Math-7b with different rollout size. Naive Scaling of Rollout Size Benefits Pass@1 , But Not Necessarily Pass@K . We present the training dynamics of Pass@1 and Pass@K performance during the RLVR training process in Figure 2 . Enlarging the rollout size allows the sampling of correct solutions to hard problems during training. We originally assumed this would benefit Pass@K performance; however, experimental results show that this is not always the case. We find that Qwen2.5-Math-7b can significantly benefit from an increased rollout size, whereas for Qwen2.5-Math-1.5b, naively scaling rollout size can even harm Pass@K performance. Figure 3: Statistical results of cumulative advantage. Both advantage calculation methods underestimate high-difficulty problems. n denotes group size. Cumulative Advantage Bias in GRPO

Read the complete text