Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning
Pranav Pawar · 2025
Read Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning by Pranav Pawar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07238.txt.
Introduction
Pranav Pawar and 5 co-authors report a tuning study in 2025 for open-source language models on math reasoning tasks. Getting the best from such models depends on settings like temperature, which controls randomness, and how many reasoning steps the model may take. The team searched these settings in a systematic way for five large models, including Qwen, Llama, DeepSeek, Mixtral, and Yi variants.
Their optimized setups cut computing cost by about 29 percent on average and sped up replies by about 24 percent, while keeping answers correct. Lower temperatures and fewer reasoning steps, four to six, helped most. DeepSeek-V3 reached the highest accuracy in their tests, while one Mixtral model gave the cheapest correct answers. One caution: every claim rests on their own benchmark runs, so other tasks may favor different settings.
Catalog record
| CREATOR | Pranav Pawar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07238
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning Pranav Pawar Abstract This paper presents a practical investigation into fine-tuning model parameters for mathematical reasoning tasks through experimenting with various configurations including randomness control, reasoning depth, and sampling strategies, careful tuning demonstrates substantial improvements in efficiency as well as performance. A holistically optimized framework is introduced for five state-of-the-art models on mathematical reasoning tasks, exhibiting significant performance boosts while maintaining solution correctness. Through systematic parameter optimization across Qwen2.5-72B, Llama-3.1-70B, DeepSeek-V3, Mixtral-8x22B, and Yi-Lightning, consistent efficiency gains are demonstrated with 100% optimization success rate. The methodology achieves an average 29.4% reduction in computational cost and 23.9% improvement in inference speed across all tested models. This framework systematically searches parameter spaces including temperature (0.1-0.5), reasoning steps (4-12), planning periods (1-4), and nucleus sampling (0.85-0.98), determining optimal configurations through testing on mathematical reasoning benchmarks. Critical findings show that lower temperature regimes (0.1-0.4) and reduced reasoning steps (4-6) consistently enhance efficiency without compromising accuracy. DeepSeek-V3 achieves the highest accuracy at 98%, while Mixtral-8x22B delivers the most cost-effective performance at 361.5 tokens per accurate response. Key contributions include: (1) the first comprehensive optimization study for five diverse SOTA models in mathematical reasoning, (2) a standardized production-oriented parameter optimization framework, (3) discovery of universal optimization trends applicable across model architectures, and (4) production-ready configurations with extensive performance characterization. 1 Introduction Large Language Models (LLMs) have shown excellent performance in math reasoning tasks but their production deployment is hampered by critical challenges related to computational efficiency, cost-effectiveness, and inference latency [ 21 ] . Although recent progress has improved model accuracy on math benchmarks significantly [ 5 ] , systematic optimization of inference hyperparameters for production deployment is mostly uncharted, which creates a large gap between research potential and realistic applicability. Mathematical reasoning is one of the most computationally intensive applications for LLMs, with models needing to be highly accurate on handling intricate multi-step problems under efficient processing [ 12 ] . The problem is further aggravated by the necessity of balancing several competing goals: solution accuracy, per-correct-answer computational expense, inference time, and system-wide throughput. Most existing methods tend to concentrate on enhancing model architectures or training methods [ 19 ] , with relatively less emphasis on post-training optimization tactics that can deliver short-term deployment gains [ 1 ] . Figure 1: Multi-Agent Optimization Framework Architecture for Systematic LLM Parameter Tuning. Shows interactions among BaseAgent, ReActAgent, AdaptivePlanner, CostTracker, ModelManager, ModelPool, and Evaluation Benchmarks. 1.1 Problem Statement and Motivation The use of LLMs for math reasoning in production settings faces a number of key bottlenecks [ 3 ] : • Cost Efficiency : Default model settings typically lead to unnecessary token use, resulting in unacceptable operational expense for high-volume usage. • Inference Latency : Math problems usually require step-by-step reasoning, which can slow down response times and make real-time use impractical [ 13 ] . • Resource Utilization : If parameters are not tuned properly, the model may use far more computational resources than necessary, this does not improve accuracy. • Model Selection : Companies often do not know which model variant is best suited to meet their particular mathematical reasoning needs. To tackle these challenges, we need a smarter way of tuning model parameters—one that takes into account how math reasoning actually works and gives clear, practical guidance for real-world use. 1.2 Research Contributions We bridge the gap between how academic models perform and how they can be reliably used in real-world applications for math reasoning [ 2 ] . The main contributions of this paper are: 1. Comprehensive Multi-Model Analysis : We conduct the first systematic optimization study of five heterogeneous SOTA models on established benchmarks for mathematical reasoning, highlighting the efficacy of each across varying architectures. 2. Production-Oriented Optimization Framework : We create and test a systematic methodology for optimizing LLM parameters in structured problem-solving tasks, achieving consistent improvements across models with clear benefits in accuracy and cost efficiency. 3. Universal Optimization Patterns : We discover and define optimization trends that generalize across model structures, such as the ubiquitous advantages of reduced temperature settings (0.1-0.4) and well-tuned reasoning step settings (4-6 steps). 4. Deployment-Ready Configurations : We offer optimal parameter settings with in-depth performance measures and explicit deployment recommendations, enabling instant adoption in real-world systems that have measurable performance requirements. 2 Related Work The rapid progress of large language models has opened up significant opportunities for Mathematical reasoning applications, yet systematically refining these models for production deployment remains an emerging field. Our study leverages recent progress across four interconnected research domains that together guide the design of our optimization framework. 2.1 Recent Advances in Making LLMs More Efficient and Optimized The rollout of ever-growing language models has increased focus on methods that align capability with computational efficiency [ 21 ] . Recent studies using advanced optimization methods indicate that thoughtful model design and training approaches can deliver significant efficiency benefits [ 4 ] . Still, these efforts target architecture over post-training parameter optimization. More closely tied to our work, recent advances [ 11 ] explore how different generation parameters shape model behavior, revealing that temperature and top-p have a strong impact on both output quality and computational efficiency. In a similar vein, studies [ 8 ] illustrate that focused parameter adjustments can lead to improvements on par with approaches that demand greater computational resources. We expand upon these results by presenting systematic optimization for problem-solving with mathematics across different model architectures. 2.2 Mathematical Reasoning Breakthroughs in Modern LLMs Mathematical reasoning has emerged as a key test for checking LLM capabilities, with several breakthrough approaches developed since 2022. The use of innovative prompting methods by Chain-of-Thought (CoT) [ 17 ] and later progress on advanced reasoning verification [ 15 ] set the stage for organized quantitative reasoning. In the latest work, advanced mathematical reasoning frameworks [ 20 ] introduced comprehensive approaches for enhanced reasoning methods, and pedagogical prompting strategies [ 16 ] expanded mathematical thinking to educational scenarios. Tool-augmented approaches have proven valuable, with recent advances in federated prompting [ 14 ] and systematic mathematical reasoning optimization [ 7 ] marked progress in numerical problem-solving through external tool integration. Still, such approaches often bring extra computational load, causing a strain between capability and efficiency that our optimization framework is designed to tackle. Recent work on theoretical foundations of CoT reasoning [ 6 ] shows similar benefits but does not provide structured parameter optimization to ensure smooth deployment. 2.3 Parameter Optimization for Faster Inference Growing interest in the systematic tuning of inference parameters has emerged as models expand in scale and deployment costs rise [ 1 ] . Recent advances offer key insights into probability tuning and generation controls, while comprehensive studies examine likelihood-driven generation strategies [ 10 ] . More recently, detailed analysis of generation parameters in various tasks [ 18 ] has been conducted, but not specifically in our research domain. Many efficiency-focused optimization strategies have been explored. Advanced architectural improvements [ 9 ] introduce betterment for attention mechanisms, while post-training compression techniques [ 11 ] demonstrate effective parameter-efficient methods. Our approach complements these methods as it focuses on optimizing parameters. This does not require any changes to the model or retraining. As