PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He · 2026
Read PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models by Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03858.txt.
Introduction
Ouyang and co-authors benchmark how well language models mix number skill with math reasoning. Numerical reasoning anchors math ability across applications. Few benchmarks join number processing and reasoning in one test. Failures in math tasks thus stay hard to interpret. PyraMathBench fills that gap with a hierarchical design.
The benchmark holds 32,505 questions drawn from 7,404 word problems. Four cognitive aspects and fourteen subcategories organize the set. Two modalities add range. Experiments expose two weak spots. Raw number computation is poor, and abstract number questions are handled weakly. Two fixes follow: SOLVE, a module for smart tool calls, and IRPO, a training method. Tool calls use fuzzy matching and reject low-quality attempts. With both fixes, one Qwen model gains five points. Math evaluation gains a diagnostic lens, not just a score.
Catalog record
| CREATOR | Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He Abstract Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench 1 1 1 https://github.com/optifine233-ship-it/PyraMathBench , a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs’ performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs’ numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training. 1 Introduction Numerical reasoning is ubiquitous in scientific research 39 , financial analysis 4 ; 17 , and integral to text understanding 51 ; 40 . Despite advances in data and compute, large language models (LLMs) like GPT-4 and Llama still struggle with math tasks 28 ; 53 , largely due to inadequate numerical processing (e.g., flawed number tokenization 20 ; 51 ) and numerical hallucination 15 ; 3 . These issues highlight the pressing need to continually enhance the numerical processing abilities of these models, as these capabilities essentially reflect the model’s proficiency in integrating numerical and mathematical reasoning, directly impacting its ability to solve real-world problems and engage in abstract thinking 45 . However, current answer-centric benchmarks fail to disentangle numerical processing flaws from mathematical reasoning errors, lacking mechanisms to diagnose failure modes related to numerical competence. While external tools show promise for boosting capabilities 32 ; 36 ; 30 , this gap also hinders performance 50 due to little interpretability. This creates an urgent need for a high-quality benchmark that enables fine-grained assessment of numerical-mathematical synergy. Current benchmarks predominantly assess the mathematical reasoning abilities of language models through math word problems (MWPs). Datasets like GSM8K 5 and APE210K 54 , based on elementary-level problems, and benchmarks such as MATH 13 , ARB 31 , and FrontierMath 8 , which involve competition-level problems like the IMO and AMC, are widely used. However, these benchmarks do not fully capture the limitations of LLMs’ capabilities. For example, when models provide incorrect answers, it remains unclear whether the failure stems from computational errors or misinterpretation of the question. Some efforts, such as LILA 26 , attempt to address this by breaking down tasks into subtasks. Akhtar et al. 1 introduced a framework to probe LLMs’ numerical reasoning at various levels. But these frameworks lack cross-task correlations, testing LLMs’ abilities in isolation without exploring how the models’ abilities to solve simpler tasks may influence their performance on more complex tasks. In fact, mathematical ability can be thought of as a hierarchy, akin to a pyramid structure, where complex tasks are broken down into simpler foundational parts. By isolating and evaluating these core tasks, we can better understand the interplay between foundational skills and higher-level reasoning. This hierarchical approach not only assesses complex tasks but also identifies how weaknesses in basic skills can affect the overall performance. Based on these, we propose the PyraMathBench (PMB), a comprehensive hierarchical benchmark providing 32,505 questions derived from 7,404 math word problems, covering 4 key cognitive aspects, 14 subcategories, and 2 modalities. Additionally, the subtasks are decomposed from real math word problems rather than generated, enhancing their relevance for real-world applications. PMB also incorporates the compositional relationships between tasks, decomposing math problems into smaller, modular subtasks that can be assessed individually and in combination. Using PMB, we evaluate a variety of state-of-the-art (SOTA) LLMs, identifying areas for improvement and offering valuable insights into the factors that influence performance. In our analysis, a key weakness observed across the LLMs is their limited capacity for abstraction, equation solving, and factual retrieval. To address this issue, we introduce the Smart Optimization & Learning-based VErsatile module (SOLVE) to enhance the flexibility and adaptability of LLMs to tool calling, as well as a new RL framework Interactive Relative Policy Optimization (IRPO). Our method yields a 5.0% improvement against the vanilla LLM. Our contributions can be summarized as: • We propose the PyraMathBench, a comprehensive hierarchical benchmark that includes 32,505 questions derived from 7,404 math word problems, covering 4 key cognitive aspects, 14 subcategories, and 2 modalities, ensuring a comprehensive evaluation. • We evaluate a variety of SOTA LLMs with this benchmark, identifying areas for improvement and offering valuable insights into the factors that influence performance. • To address the challenges of mathematical reasoning, we introduce SOLVE module and IRPO algorithm, designed to mitigate tool call parsing failures and improve the efficiency of tool calling. Figure 1: Taxonomy of PyraMathBench and two examples of decomposing complex reasoning problems (green and pink) into subtasks, dashed lines representing multimodal tasks. 2 The Taxonomy of PyraMathBench The core motivation behind PMB’s taxonomy is the recognition that mathematical tasks often require multiple layers of cognitive aspects and computational skills, ranging from simple numerical parsing to intricate logical reasoning. An LLM’s ability to solve a high-level math word problem is contingent upon its proficiency in handling lower-level subcomponents. Previous benchmarks lack an explicit framework for isolating different cognitive aspects, making it difficult to diagnose specific failure points. By decomposing complex mathematical tasks into distinct hierarchical aspects, PMB provides a systematic method to evaluate capability at each stage of mathematical cognition, allowing for a more interpretable assessment of LLM performance. Inspired by previous research 46 ; 1 and Piaget’s cognitive theory, taking into account both the reasoning paradigm and the feasibility of annotation, our benchmark taxonomizes tasks into four hierarchical aspects (A1–A4), encompassing 14 distinct tasks. Figure 2 shows the composition of subtasks at each aspect and examples of subtask annotation. Complex Reasoning evaluates the model’s ability to integrate multiple cognitive processes and mathematical principles. It requires sophisticated logical deductions, image interpretation, and multistep problem-solving. Understanding focuses on the model’s ability to comprehend and interpret mathematical content, transforming unstructured text into mathematical representations. Calculation primarily tests computational efficiency and correctness in solving mathematical problems. Finally, Numerical Parsing assesses the model’s ability to parse and process numerical data in various formats, thereby evaluating its capacity to recognize, interpret, and extract numerical information for further computation. The specific descriptions, prompts, and examples for 14 subtasks are provided in Appendix A . Figure 2: Data distribution of 14 subtasks. 3 Construction and Statistics Data Sources. The PMB dataset incorporates six existing evaluation datasets and practice question collections. The data collection adheres to the following guidelines: 1) It includes common mathematical problems and visual reasoning tasks to represent the typical problem distribution. 2) Each problem is structured to allow clear decomposition into subtasks, facilitating unambiguous labeling. 3) The dataset is varied in difficulty, ensuring the inclusion of challenging tasks to effectively evaluate the performance of LLMs. We excluded non-mathematical content from the data. Based on this, we considered 6 datasets as data sources: ASDiv 25 , alg514 18 , Dolphin 18K 37 , SVAMP 28 ,