Decomposing Elements of Problem Solving: What "Math" Does RL Teach?
Tian Qin, Core Francisco Park, Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, David Alvarez-Melis · 2025
Read Decomposing Elements of Problem Solving: What "Math" Does RL Teach? by Tian Qin, Core Francisco Park, Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, David Alvarez-Melis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22756.txt.
Introduction
Tian Qin and co-authors break math problem solving into parts to see what reinforcement learning teaches in this 2025 paper. Training methods have raised language model scores on math tests, but a score alone cannot show which skill improved. The authors split solving into planning the steps, executing them, and verifying the result.
Tests show the training mainly sharpens execution: models get more reliable on problems they already know how to approach. On truly new problems they stall, a coverage wall blamed on weak planning. A small synthetic task that mimics solution trees reproduced the pattern. In that setting, the authors find conditions where training can break through the wall by exploring new paths.
Catalog record
| CREATOR | Tian Qin, Core Francisco Park, Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, David Alvarez-Melis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decomposing Elements of Problem Solving: What "Math" Does RL Teach? Tian Qin, Core Francisco Park, Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, David Alvarez-Melis Abstract Mathematical reasoning tasks have become prominent benchmarks for assessing the reasoning capabilities of LLMs, especially with reinforcement learning (RL) methods such as GRPO showing significant performance gains. However, accuracy metrics alone do not support fine-grained assessment of capabilities and fail to reveal which problem-solving skills have been internalized. To better understand these capabilities, we propose to decompose problem solving into fundamental capabilities: Plan (mapping questions to sequences of steps), Execute (correctly performing solution steps), and Verify (identifying the correctness of a solution). Empirically, we find that GRPO mainly enhances the execution skill—improving execution robustness on problems the model already knows how to solve—a phenomenon we call temperature distillation . More importantly, we show that RL-trained models struggle with fundamentally new problems, hitting a ‘coverage wall’ due to insufficient planning skills. To explore RL’s impact more deeply, we construct a minimal, synthetic solution-tree navigation task as an analogy for mathematical problem-solving. This controlled setup replicates our empirical findings, confirming RL primarily boosts execution robustness. Importantly, in this setting, we identify conditions under which RL can potentially overcome the coverage wall through improved exploration and generalization to new solution paths. Our findings provide insights into the role of RL in enhancing LLM reasoning, expose key limitations, and suggest a path toward overcoming these barriers. Code is available at https://github.com/cfpark00/RL-Wall . 1 Harvard University 2 Kempner Institute 1 Introduction Figure 1 : Decomposing problem solving and modeling the effects of RL in a toy setting. (a): We decompose math problem solving into fundamental capabilities. To solve a problem model must first plan (map the question to a sequence of actions) and then execute (navigate the tree) without errors. (b): We design a minimal toy environment that explicitly captures these core reasoning elements in a controlled setting. (c): GRPO primarily improves precision on problems the model already partially solves, without expanding overall coverage . Our synthetic setup reproduces this behavior—and crucially, shows that under the right conditions, RL can break the coverage wall. Recent months have seen rapid gains in LLM mathematical reasoning, driven in large part by reinforcement learning (RL) methods like the simple yet effective GRPO [ 1 ] . This progress has largely been benchmarked using a single evaluation metric: Pass@1 accuracy—the fraction of problems for which the model produces a correct solution on its first attempt. This mirrors a long-standing tendency in human cognition research to rely on simple, aggregate metrics—such as standardized test scores [ 2 , 3 ] or IQ measures [ 4 ] —to evaluate complex capabilities. These metrics persist in part because they are easy to interpret and compare across individuals [ 5 , 6 ] . While convenient, relying on a single accuracy score like Pass@1 to evaluate LLM reasoning has two important limitations. First, it is monolithic : too coarse-grained to reveal the nature of model failures or the specific reasoning skills being improved, such as whether RL enhances planning, computation, error-checking, or merely reinforces existing knowledge. Second, it evaluates greedy decoding, whereas real-world deployments—especially those requiring complex reasoning—often rely on stochastic sampling (e.g., temperature decoding, best-of-N selection) to increase diversity or robustness. To truly understand the impact of RL on reasoning, we need evaluation methods that go beyond aggregate accuracy and capture how models navigate the problem-solving process. To address these limitations, we adopt a finer-grained evaluation framework aimed at understanding two key questions: (i) are the improvements from GRPO uniform across problems?, and (ii) what specific aspects of reasoning does GRPO improve? By breaking down accuracy per problem, we find that GRPO is most effective on problems the model already knows how to solve (Section 3 ). On these problems, its effect is to make the correct solution more likely regardless of sampling temperature—a phenomenon we call temperature distillation . This enhances execution robustness under sampling but does little to help with fundamentally novel problems, exposing a coverage wall , defined as the limit of Pass@ K as K\rightarrow\infty (Fig. 1 c ). To study (ii) we introduce a more fine-grained framework for analyzing reasoning, viewing problem solving as a search process in a solution tree (Section 4 ). This perspective decomposes reasoning into three core skills (Fig. 1 a ): Plan (mapping the question to a sequence of actions indicating where to go in the tree, Execute (carrying out those steps correctly by performing the navigation), and Verify (detecting and correcting errors). Using this lens, we find that even small models often plan well but frequently fail in execution, making logical and factual errors. GRPO, we show, primarily strengthens execution. To better isolate and understand RL’s effects, we construct a minimal, synthetic setting that abstracts mathematical problem solving. (Fig. 1 b , Sec. 5 ). Here, the model needs to map the question to a sequence of actions (i.e., plan) and execute the sequence of actions to provide the correct answer. This controlled environment closely mirrors our proposed framework, while allowing control over the underlying data generation process. We find that this synthetic setting faithfully reproduces our empirical observations. Crucially, the setup also reveals properties of the pretraining and RL data which allows RL to break through the coverage wall—namely, when the model is able to explore new actions and apply them to structurally similar problems. Our study provides insights into RL’s role in enhancing LLM reasoning. Our contributions are: • Reasoning Decomposition: We propose a framework that decomposes math problem solving into three capabilities: planning, execution, and verification. • Empirical Analysis of RL: In the proposed decomposition, we show that GRPO primarily improves execution on known problems, leading to a temperature distillation effect. However, GRPO fails to help model solve previously unsolved problems, revealing a coverage wall. • Synthetic Validation: We construct a minimal synthetic task that closely reflects the fundamental elements of problem solving and replicates empirical findings. We reveal some data properties under which RL can overcome the coverage wall. 2 Related Work 2.1 Math Reasoning Benchmarks Mathematics has long been a core domain for evaluating AI reasoning, dating back to early work on algebra word problems and logic inference [ 7 , 8 , 9 , 10 ] . Recently, math word problems involving basic arithmetic and multi-step reasoning such as GSM8K [ 11 ] and MATH [ 12 ] has become standard benchmarks for elementary and high school–level tasks [ 13 ] . As model performance has saturated on many of these benchmarks, recent efforts have pushed toward more challenging datasets include olympiad- and competition-level problems [ 14 , 15 , 16 ] . Outside math word problems, the reasoning capabilities have been extended to theorem proving benchmarks [ 17 , 18 ] as well as multi-modal problems [ 19 , 20 , 21 ] . While benchmark complexity has increased, relatively little work has focused on understanding what these benchmarks actually measure about LLM reasoning—and which skills drive model performance improvements. 2.2 Reinforcement Learning for LLM Post Training Post-training methods for improving LLM reasoning generally fall into three categories: supervised fine-tuning [ 22 , 23 ] ,