How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair PubID: pubid: 979-8-3315-9147-2/25/$31.00 © 2025 IEEE

Kazuki Kusama 1 , Honglin Shu 1 , Masanari Kondo 1 , Yasutaka Kamei 1 · 2025

Read How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair PubID: pubid: 979-8-3315-9147-2/25/$31.00 © 2025 IEEE by Kazuki Kusama 1 , Honglin Shu 1 , Masanari Kondo 1 , Yasutaka Kamei 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16499.txt.

Introduction

Kazuki Kusama and co-authors ask how small a language model can be while still fixing code well, in a study from 2025. Large models fix bugs with high accuracy but need a lot of computing power. Small models run on far less, yet their skill at repair was less clear.

The team tested small and large models on a known set of buggy programs called QuixBugs. The newest small models fixed bugs as well as the large ones, and in some cases better. They also tested a method that stores the model in fewer bits. That step cut memory use a great deal with little loss in accuracy. The authors conclude that small models are a real option for repair tools.

Catalog record

CREATORKazuki Kusama 1 , Honglin Shu 1 , Masanari Kondo 1 , Yasutaka Kamei 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16499

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair PubID: pubid: 979-8-3315-9147-2/25/$31.00 © 2025 IEEE Kazuki Kusama 1 , Honglin Shu 1 , Masanari Kondo 1 , Yasutaka Kamei 1 Abstract Background: Large language models (LLMs) have greatly improved the accuracy of automated program repair (APR) methods. However, LLMs are constrained by high computational resource requirements. Aims: We focus on small language models (SLMs), which perform well even with limited computational resources compared to LLMs. We aim to evaluate whether SLMs can achieve competitive performance in APR tasks. Method: We conducted experiments on the QuixBugs benchmark to compare the bug-fixing accuracy of SLMs and LLMs. We also analyzed the impact of int8 quantization on APR performance. Results: The latest SLMs can fix bugs as accurately as—or even more accurately than—LLMs. Also, int8 quantization had minimal effect on APR accuracy while significantly reducing memory requirements. Conclusions: SLMs present a viable alternative to LLMs for APR, offering competitive accuracy with lower computational costs, and quantization can further enhance their efficiency without compromising effectiveness. I Introduction Debugging software systems, a costly process in development [ 1 ] , must be performed efficiently to reduce costs while identifying and fixing bugs. To that end, Automated Program Repair (APR) has been widely studied as a technique to automate the debugging process [ 2 , 3 ] . APR can automatically identify program bugs and generate patches to fix them [ 4 ] . Since 2022, Large Language Model (LLM)-based APR has emerged [ 5 ] , achieving significantly higher accuracy than existing APR methods [ 6 , 7 ] . Unlike traditional APR methods trained on a limited number of datasets, LLMs are trained on vast corpora containing up to hundreds of billions of text and code tokens, enabling them to fix even complex bugs. Despite the high accuracy of LLM-based APR, it requires substantial computational resources [ 8 ] . For example, Code Llama-70B, which demonstrated high accuracy in APR [ 9 ] , contains 70 billion parameters and requires at least two NVIDIA A100 80GB GPUs for inference. Such a requirement makes it impractical for developers to use these models in their standard development environments, such as on personal laptops. Since debugging is a fundamental process in development and is executed frequently by developers, reducing the cost of debugging is an essential new challenge. To remedy this challenge, APR can use Small Language Models (SLMs) and quantization techniques . SLM is also a language model, but it has significantly fewer parameters than LLM, reducing computational and memory requirements for inference and training. Quantization converts model weights to lower-bit representations (e.g., converting from float32 to int8), reducing model size and memory usage. While this conversion typically results in some accuracy loss, quantized models can achieve faster inference and lower latency. If SLMs and quantization techniques can effectively reduce computational costs while keeping repair accuracy loss to negligible levels, they will enable the practical deployment of LLM-based APR in standard software development environments. In this paper, we explore 14 SLMs and 2 LLMs, both widely used in APR tasks [ 10 , 11 , 12 ] . Secondly, we investigate quantization techniques with four bit representations (float32, float16, int8, and int4). Through an experiment with QuixBugs-Python and QuixBugs-Java [ 13 ] , we address the following research questions (RQs): RQ1: How well can Small Language Models fix bugs? RQ2: How does quantization affect bug fixes? Our experiments showed that the best-performing SLMs, such as Phi-3 (3.8B), successfully fixed up to 38 out of 40 bugs, comparable to the performance of the best-performing LLM, Codex, which fixed 39 out of 40 bugs. We also found that int8 quantization had a minimal impact on repair accuracy, with differences of approximately +0.5 bugs compared to float32, the maximum bit representation. Our main contributions are as follows: (1) This is the first comprehensive evaluation of 14 SLMs in APR tasks. We compare the performance of SLMs with LLMs and provide practical insights into the capabilities and limitations of SLMs. (2) We evaluate the impact of quantization on APR performance, demonstrating that int8 quantization offers a practical trade-off between efficiency and accuracy. (3) We provide a practical recommendation: code-specific SLMs with int8 quantization can achieve performance comparable to that of LLM-based APR methods, while requiring fewer computational resources. (4) We provide a comprehensive replication package, including all experimental data and code, to facilitate further research. 1 1 1 https://doi.org/10.5281/zenodo.15472061 II Background This section provides an overview of APR methods, LLM-based APR methods as well as their challenges. Additionally, we introduce SLMs and quantization techniques as potential alternatives to LLMs. II-A APR APR methods aim to automate debugging processes. APR methods first identify the buggy location and fix the source code to pass the test cases. There exist various APR methods, including heuristic-based [ 14 ] , constraint-based [ 15 ] , tempalate-based [ 16 ] , and learning-based approaches [ 17 , 18 ] . For example, learning-based approaches are based on machine learning techniques such as neural machine translation (NMT) [ 17 , 18 ] . These methods are trained on bug-fix examples. Once trained, they can provide modifications to the buggy code. With the rapid development of machine learning technologies, learning-based APR methods have become the most popular approach. Specifically, LLM-based APR methods attract attention due to their high accuracy and flexibility. OpenAI Codex [ 19 ] , for instance, has demonstrated substantial improvements over state-of-the-art APR techniques [ 5 , 6 ] . While AlphaRepair, the most accurate traditional APR method, achieved around 26% accuracy, Codex significantly outperformed it with approximately 38% accuracy [ 5 ] . This highlights the effectiveness of LLM-based approaches in APR tasks, as they can generate high-quality patches for a diverse range of buggy code without requiring additional fine-tuning. II-B Efficiency Challenges of LLM-Based APR LLM-based APR methods achieve high accuracy; however, they face non-functional challenges [ 8 , 20 ] . Yang et al. [ 8 ] investigated 146 papers on code-related tasks using LLMs and identified six important non-functional properties: robustness, security, privacy, explainability, efficiency, and usability. In this study, we focus on the challenges of efficiency since they lead to difficulties in applying LLM-based APR methods to a practical software development workflow. For example, as an efficiency challenge, Yang et al. [ 8 ] stated that integrating LLM-based approaches into a standard software development workflow is difficult due to the large size of LLMs. While a model size of 50 MB is considered feasible for integration into modern software development tools such as an integrated development environment (IDE) [ 21 ] , current LLMs far exceed this threshold. For example, CodeBERT has 125 million parameters and occupies 560 MB of space. 2 2 2 This value is estimated using the Hugging Face memory estimator [ 22 ] . Such large models lead to high memory consumption and long inference latency, making it difficult for individual developers to use these models in their own environment. II-C Small Language Models (SLMs) To address the efficiency challenges of LLMs, reducing the model size is a promising approach. SLMs therefore emerged as a potential alternative to LLMs.

Read the complete text