Mathematical Computation and Reasoning Errors by Large Language Models
Liang Zhang, Edith Aurora Graf · 2025
Read Mathematical Computation and Reasoning Errors by Large Language Models by Liang Zhang, Edith Aurora Graf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09932.txt.
Introduction
Liang Zhang and Edith Aurora Graf test four large language models on math tasks made to trip them, and log the step where each slip occurs. Their paper is from two thousand twenty-five.
Such models now teach and grade math, so their care with facts and steps is a base need. The team built tasks in three fields — sums, algebra, and number theory — from item models set to be hard and error-prone, in place of stock test sets. They grade both the end answer and each step of the work. Four known models are tried. The aim is a clear map of where model math goes wrong, so that school use can rest on proof, not trust.
Catalog record
| CREATOR | Liang Zhang, Edith Aurora Graf |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-09932
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mathematical Computation and Reasoning Errors by Large Language Models Thanks: This paper has been officially accepted for presentation at the Artificial Intelligence in Measurement and Education Conference (AIME-Con) 2025, held in Pittsburgh, Pennsylvania, United States, from October 27 to 29, 2025. Liang Zhang, Edith Aurora Graf Abstract Large Language Models (LLMs) are increasingly utilized in AI-driven educational instruction and assessment, particularly within mathematics education. The capability of LLMs to generate accurate answers and detailed solutions for math problem-solving tasks is foundational for ensuring reliable and precise feedback and assessment in math education practices. Our study focuses on evaluating the accuracy of four LLMs (OpenAI GPT-4o and o1, DeepSeek-V3 and DeepSeek-R1) solving three categories of math tasks, including arithmetic, algebra, and number theory, and identifies step-level reasoning errors within their solutions. Instead of relying on standard benchmarks, we intentionally build math tasks (via item models) that are challenging for LLMs and prone to errors. The accuracy of final answers and the presence of errors in individual solution steps were systematically analyzed and coded. Both single-agent and dual-agent configurations were tested. It is observed that the reasoning-enhanced OpenAI o1 model consistently achieved higher or nearly perfect accuracy across all three math task categories. Analysis of errors revealed that procedural slips were the most frequent and significantly impacted overall performance, while conceptual misunderstandings were less frequent. Deploying dual-agent configurations substantially improved overall performance. These findings offer actionable insights into enhancing LLM performance and underscore effective strategies for integrating LLMs into mathematics education, thereby advancing AI-driven instructional practices and assessment precision. 1 Introduction Large Language Models (LLMs) have significantly impacted mathematical instruction and assessment. Educational platforms are increasingly integrating LLMs to enhance teaching and evaluation methods. For instance, Khan Academy utilized the LLM-powered tool Khanmigo for Socratic-style math assistance Anand 2023 . Coursera uses LLMs to streamline assessment creation, automate grading, and offer personalized feedback Maggioncalda 2024 . Quizlet’s Q-Chat integrates LLM-based conversational AI to dynamically adapt question difficulty levels and deliver guided hints Bayer 2025 . Beyond merely producing final answers, LLMs excel at clearly articulating intermediate computational steps and reasoning processes, significantly enhancing their value in mathematics education contexts Gupta et al. 2025 . These capabilities of LLMs facilitate personalized tutoring, interactive problem-solving, and real-time feedback, significantly reduce grading workloads and ensure consistent evaluations in mathematics education. The capability of LLMs to produce accurate answers and detailed, step-by-step solutions in math problem-solving is foundational for reliable assessment and precise feedback in mathematics education Gupta et al. 2025 ; Jin et al. 2025 . Specifically, LLM-based automated assessment involves evaluating granular math skills through step-level grading of student solutions Jin et al. 2025 , performing automatic step-level corrections Li et al. 2025 , and providing targeted instructional hints Tonga et al. 2025 . However, this raises an important question: if LLMs cannot reliably produce correct answers or accurately solve math problems, can their outputs still be considered effective and trustworthy for instructional guidance and learner assessment? This motivated our idea to systematically test the capability of LLMs to accurately solve diverse math tasks, and subsequently extend their applicability toward realistic assessment and instructional scenarios. While state-of-the-art LLMs have demonstrated high accuracy on various math benchmarks, benchmark success alone does not present a comprehensive picture. Performance significantly declines on certain math tasks like fundamental numerical understanding and basic computational problems Yang et al. 2024 ; Boye and Moell 2025 ; Petrov et al. 2025 . Current LLM computation and reasoning processes remain prone to calculation mistakes (e.g., arithmetic slips, algebraic simplification errors) and logical reasoning errors (e.g., invalid inference steps, omission of necessary procedural steps, and self-contradictory reasoning) Li et al. 2024 ; Roy et al. 2025 . These persistent errors significantly limit the reliability and efficacy of LLM outputs for instructional feedback and learner assessment purposes. In this study, we specifically examine the capabilities and limitations of LLMs in math problem-solving, focusing on assessing the accuracy of generated answers and identifying errors within solution steps. Instead of using benchmarks, we built math problems in arithmetic, algebra, and number theory to evaluate LLMs’ proficieny in math computation and reasoning. We explored four distinct LLMs, two base models GPT-4o Hurst et al. 2024 and DeepSeek-V3 Liu et al. 2024 and two reasoning-enhanced models OpenAI o1 Jaech et al. 2024 and DeepSeek-R1 Guo et al. 2025 , across three math problem-solving tasks, including arithmetic, algebra, and number theory. Two interaction paradigms are considered: (i) a single-agent setting in which one model works through each math task step-by-step, and (ii) a dual-agent setting that lets two peer LLMs chat, cross-validate, and refine their reasoning, echoing recent advances in collaborative intelligence Zhang et al. 2025a ; Latif et al. 2024 ; Zhang et al. 2025b . Every solution was decomposed into granular steps and coded with an expert-verified rubric, which enabled us to quantify step-level accuracy and localize procedural or conceptual errors. Our investigation is guided by two R esearch Q uestions: Q1: How accurately do LLMs generate final answers to math problems? Q2: What recurring error patterns (procedural, conceptual, or logical) emerge in their step-level solutions? This study will provide researchers and practitioners with precise insight into where LLMs excel and where they falter in math computation and reasoning. We also provide a rubric that can be used to evaluate the accuracy of LLM-based solutions, and to identify the nature of errors when they occur. Our study demonstrates that knowledgeable LLMs have potential to reliably support math instruction and assessment, and we offer actionable guidance for their effective use. 2 Related Works LLMs have made notable strides in solving math problems, yet they frequently struggle with precise computations and multi-step numerical reasoning tasks Wolfram 2023 ; Li et al. 2024 . Frieder et al. Frieder et al. 2023 provided a detailed evaluation demonstrating that GPT-4 (the 2023 version of ChatGPT) effectively handles many undergraduate-level questions but exhibits significant difficulties when confronted with graduate-level math challenges, particularly in proof-based tasks and complex symbolic computations. Yang et al. Yang et al. 2024 found that LLMs frequently make surprising mistakes in basic numerical understanding and processing tasks. Some studies Li et al. 2024 ; Pan et al. 2025 have identified recurring error patterns in math reasoning by LLMs, including calculation errors, counting errors, formula confusion, question misinterpretation, missing solution steps, conceptual confusion, and nonsensical outputs, among others. Numerous additional studies have consistently reported similar challenges, emphasizing the ongoing limitations faced by LLMs in performing math tasks Arkoudas 2023 ; Wolfram 2023 ; Wang et al. 2023 ; Zhang et al. 2024 ; McLeish et al. 2024 . Mitigating math problem-solving errors and boosting LLM performance is a multifaceted endeavor. Technical advances center on modular reasoning strategies, such as Chain-of-Thought and Program-of-Thought prompting Wei et al. 2022 ; Chen et al. 2022 , alongside fine-tuning and math-specific training regimens Zhang et al. 2023 ; Ahn et al. 2024 , novel architectures that integrate external tools or structured reasoning modules, and rigorous evaluations that precisely expose weaknesses. On the usability side, carefully designed prompts, curriculum-aligned task sequencing, interactive dialogue, and built-in self-checking routines can substantially reduce errors in real-world use. 3 Methods Dataset. For the dataset, we used the problem categories and