Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
Afrar JahinSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Arif Hassan ZidanSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Wei ZhangSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA w · 2025
Read Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach by Afrar JahinSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Arif Hassan ZidanSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Wei ZhangSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA w (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10573.txt.
Introduction
Afrar Jahin and four co-authors grade the math of language models in twenty twenty-five. Mathematical reasoning needs multi-step logic and abstract generalization. It remains one of the hardest skills for large language models. Prior evaluations were often narrow, covering few models or one benchmark.
This study compares eight leading models, including two recent DeepSeek systems. Three independent benchmark data sets supply the tests. The analysis works at fine grain, splitting performance by domain. DeepSeek-R1 performs competitively with OpenAI's o1 across most domains. It achieves the highest accuracy on the MMLU formal logic benchmark. Readers get a detailed map of where today's models reason well in mathematics, and where they still fail.
Catalog record
| CREATOR | Afrar JahinSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Arif Hassan ZidanSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Wei ZhangSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA w |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10573
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach Afrar JahinSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Arif Hassan ZidanSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected], Wei ZhangSchool of Computer and Cyber SciencesAugusta University, Augusta, GA, USA [email protected] Yu BaoDepartment of Graduate PsychologyJames Madison University, Harrisonburg, VA, USA [email protected] Tianming LiuSchool of ComputingUniversity of Georgia, Athens, GA, USA [email protected] Abstract With the rapid advancement of Artificial Intelligence (AI), Large Language Models (LLMs) have significantly impacted a wide array of domains, including healthcare, engineering, science, education, and mathematical reasoning. Among these, mathematical reasoning remains a particularly challenging capability, often requiring multi-step logic and abstract generalization. While prior work has explored LLM performance on reasoning tasks, comprehensive evaluations that span both depth and breadth across model families remain limited. In this study, we present a systematic evaluation of mathematical reasoning abilities across eight leading LLMs, including two recent DeepSeek models, using three independent benchmark datasets. Our analyses reveal several key findings: (1) DeepSeek-R1 performs competitively with o1 across most domains and achieves the highest accuracy on the MMLU Formal Logic benchmark; (2) distilled variants, such as DeepSeek-1.5B, exhibit substantial performance degradation; and (3) Gemini 2.0 Flash achieves the lowest response latency. Beyond quantitative metrics, we explore how architectural choices, training paradigms, and optimization strategies contribute to variation in reasoning performance. These findings provide new insights into the capabilities and limitations of current LLMs in mathematical domains, and offer guidance for the development of future models better aligned with rigorous reasoning demands. 1 Introduction As Artificial Intelligence (AI) continues to advance at an unprecedented pace, an array of powerful Large Language Models (LLMs) has emerged, including OpenAI’s GPT-4o, o1, o3, Claude 3.5, Llama 4, Qwen 3, Gemini 2.0 and so on [ Achiam et al. 2023 , Chang et al. 2024 , Dubey et al. 2024 , Enis and Hopkins 2024 , Kasneci et al. 2023 , Liu et al. 2024b , Team et al. 2024 , Zhao et al. 2023 ] . Driven by cutting-edge advancements in Deep Neural Networks (DNNs), these models integrate human cognition elements to enhance problem-solving and decision-making [ Lappin 2024 ] . Serving as a guiding light in natural language processing (NLP) [ Min et al. 2023 ] , healthcare [ Dai et al. 2023 , Liu et al. 2023c , Liu et al. 2023d ] , clinical textual processing [ Dai et al. 2025 , Liu et al. 2023b , Zhong et al. 2023 ] , biomedical image analyses [ Liu et al. 2023c ] , code generation [ Liu et al. 2023a ] , decision support [ Li et al. 2022 ] , multimodal data analytics [ Wang et al. 2024 , Xiao et al. 2024 ] , and mathematical reasoning [ Ahn et al. 2024a , Li et al. 2024b , Peng et al. 2024 , Shao et al. 2024 , Xin et al. 2024 , Zhong et al. 2024 , Zhuang et al. 2024 ] , LLMs push the boundaries of AI capabilities, approximating human-like reasoning through sophisticated statistical inference [ Achiam et al. 2023 , Kasneci et al. 2023 , Neha and Bhati 2025 ] . However, despite their transformative potential, these models face notable limitations. Their high computational demands pose significant barriers to broader accessibility, making large-scale implementation costly [ Neha and Bhati 2025 ] . Moreover, while LLMs perform well in general contexts, they often struggle with specialized tasks, exhibiting inconsistencies in performance. Multimodal models, for example, continue to face challenges in spatial reasoning and real-world physics, while AI-assisted code generation frequently produces syntactically correct yet functionally flawed outputs, requiring human oversight [ Liu et al. 2023a , Zhong et al. 2024 ] . These constraints underscore the ongoing need for refinement and innovation in AI research to bridge the gap between artificial and human intelligence. In 2024, OpenAI released GPT-4o, a multimodal model capable of processing text, images, and audio. With estimated 2 trillion parameters, it significantly surpasses GPT-3 in tasks such as reasoning, coding, and clinical diagnostics [ Achiam et al. 2023 , Dale 2021 , Hurst et al. 2024 , Shahriar et al. 2024 ] . Meanwhile, other cutting-edge models, such as o1, o3 and recently o4-mini, have been introduced between 2024 and 2025. In particular, o1 enhances reasoning capabilities by incorporating a Chain-of-Thought (CoT) approach [ Stechly et al. 2025 , Zhong et al. 2024 ] , where it renders intermediate reasoning steps before reaching a final answer closely mirroring the way humans process complex problems [ Arrieta et al. 2025 ] . In addition, Claude 3.5, released in 2024, emphasizes safety, alignment, and performance, with improvements in reasoning, language understanding, and handling complex tasks like text and code generation [ Enis and Hopkins 2024 ] . With 250B parameters, it surpasses earlier models in accuracy and ethical alignment. It supports up to 200K tokens for extended context. Notably, enhanced by reinforcement learning from human feedback (RLHF) [ Hu et al. 2024 , Wen et al. 2024 ] and Constitutional AI, it reduces undesirable responses, biases, and better aligns with human intent. Claude 3.5 excels in specialized areas like coding and scientific reasoning, with improved transparency and ethical safeguards [ Enis and Hopkins 2024 , Neha and Bhati 2025 ] . Furthermore, Llama-3.3, presented in 2024, is the latest version of LLM in Meta AI family, following Llama-1 and Llama-2. Llama-3.3 advances further with 70B parameters and 128K token context window, improved by grouped-query attention for better efficiency [ Dubey et al. 2024 , Lu et al. 2024 ] . Llama 3.1 excels in coding, logical problem solving, and low-resource language tasks. Unlike closed models such as GPT series, it remains open-weight and freely accessible for research and commercial use, but is restricted to text-only input [ Dominguez-Olmedo et al. 2025 , Lu et al. 2024 , Neha and Bhati 2025 ] . Moreover, Gemini 2.0 Flash, initially introduced as an experimental variant, provides significant speed and efficiency gains over its predecessor, Gemini 1.5 Flash, without sacrificing the efficiency [ Huang and Zhang 2024 , Imran and Almusharraf 2024 , Team et al. 2024 ] . Notably, it outperforms Gemini 1.5 Pro on key benchmarks while operating at twice the speed. It enables the incorporation of agentic AI and native use, allowing the model to call external functions (Google Search and Maps) and integrate streaming data for expanded real-time applications. By combining better performance in tasks such as math, code generation, and multilingual audio output with enhanced efficiency. Gemini 2.0 aims to deliver comprehensive, cost-effective AI solutions for both developers and end users [ Huang and Zhang 2024 , Team et al. 2024 ] . Lastly, Qwen2.5, released in September 2024, is the latest iteration in the Qwen series, following Qwen2 in June 2024 and the original Qwen in August 2023. Qwen1.5 featured models up to 72B parameters, emphasizing efficiency and open-source accessibility. Qwen2 introduced improved reasoning, multilingual support, and coding capabilities, with models scaling up to 72.71B parameters [ Yang et al. 2024 , Zhu et al. 2025 ] . Besides, established in 2023 as a research initiative to push the boundaries of artificial general intelligence