On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study CCS: Software and its engineering Software development methods CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Machine learning
Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto · 2026
Read On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study CCS: Software and its engineering Software development methods CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Machine learning by Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23059.txt.
Introduction
Antonio Vitale and three co-authors test whether code comments help machines fix bugs. Automated bug fixing turns a broken method into a corrected one with language models. A common prep step strips comments from code before training, on the assumption that they add noise.
Their empirical study checks that assumption directly. If comments guide fixes, removing them wastes a useful signal about intent. The answer shapes how training data for coding models should be cleaned across the field. Removing comments before training is standard practice, so the study's verdict could change preprocessing habits across the field. The work is empirical throughout, measuring real fixing performance with and without the comments present in code.
Catalog record
| CREATOR | Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study CCS: Software and its engineering Software development methods CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Machine learning Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto Abstract . Large Language Models (LLMs) are increasingly relevant in Software Engineering research and practice, with Automated Bug Fixing (ABF) being one of their key applications. ABF involves transforming a buggy method into its fixed equivalent. A common preprocessing step in ABF involves removing comments from code prior to training. However, we hypothesize that comments may play a critical role in fixing certain types of bugs by providing valuable design and implementation insights. In this study, we investigate how the presence or absence of comments, both during training and at inference time, impacts the bug-fixing capabilities of LLMs. We conduct an empirical evaluation comparing two model families, each evaluated under all combinations of training and inference conditions ( with and without comments), and thereby revisiting the common practice of removing comments during training. To address the limited availability of comments in state-of-the-art datasets, we use an LLM to automatically generate comments for methods lacking them. Our findings show that comments improve ABF accuracy by up to threefold when present in both phases, while training with comments does not degrade performance when instances lack them. Additionally, an interpretability analysis identifies that comments detailing method implementation are particularly effective in aiding LLMs to fix bugs accurately. 1. Introduction Large Language Models (LLMs), are increasingly becoming indispensable tools for assisting software developers in their daily tasks. These models enable the automation of a wide range of activities. Furthermore, developers can easily leverage advanced LLM-powered tools, such as GitHub Copilot, which are seamlessly integrated into modern Integrated Development Environments (IDEs) to enhance productivity and streamline workflows. Among these applications, Automated Bug Fixing (ABF) has emerged as one of the most researched and impactful coding tasks ( 56 ; 38 ; 4 ; 33 ; 9 ; 68 ; 25 ) . ABF involves identifying and resolving bugs within a method ( e.g., a scenario where a test case fails). While the concept may seem straightforward, ABF holds significant value for developers, as bug fixing is notoriously challenging and often demands considerable effort ( 14 ; 45 ) . By alleviating this burden, ABF has the potential to substantially enhance developer efficiency and reduce the time spent on error resolution. Early research on using deep learning (DL) and LLMs for ABF established the foundation for the field. Tufano et al. ( 56 ) presented one of the first DL-based approaches to automatically fix bugs, quickly followed by several others ( 9 ; 33 ; 68 ) . A key contribution of the work by Tufano et al. ( 56 ) was the release of a benchmark dataset, which remains a standard for training and evaluating ABF models ( 63 ; 38 ; 7 ; 13 ; 65 ; 2 ; 18 ; 22 ; 23 ) . Such a dataset, mined from GitHub, contains couples of buggy Java methods and their respective fixed versions. Given the limitations of the specific ML model they adopted (Recurrent Neural Networks — RNNs), they needed to pre-process the source code of the methods so that identifiers are abstracted ( i.e., they appear like IDi , where i is an incremental number) and comments are removed. However, code comments may not only be important ( 61 ) , but often essential for fixing bugs. Comments, indeed, might contain precious information about the requirements, which are sometimes necessary to understand what is wrong with the code and, thus to fix it. Let us consider the example in Fig. 1 . The buggy code lacks mNearbyDeviceAdapter.notifyDataSetChanged() . Such an instruction notifies the dataset change to the nearby devices. Without reading the comment and understanding this requirement, neither a human developer nor a model could identify or fix the bug. Figure 1. Example of a buggy method and its fixed version (the green line is the fix, which was missing in the buggy version). Unlike the RNNs employed by Tufano et al. ( 56 ) , modern LLMs are well-equipped to process code comments. Despite this capability, prior research has overlooked the potential value of incorporating comments—a potentially critical source of information—for Automated Bug Fixing (ABF). In this work, we investigate how the presence or absence of code comments, both during training and at inference time, impacts on the bug-fixing capabilities of Large Language Models (LLMs). First, we build a new dataset for ABF based on the one introduced by Tufano et al. ( 56 ) . To do this, we consider the same repositories and commits, but we re-extract the instances by considering the raw source code ( i.e., we do not abstract identifiers and keep the original comments). Differently from the previous work, which only considered small - ( \leq 50 tokens) and medium -sized ( \leq 100 tokens) methods in two distinct datasets, we merge them in a single dataset, in which we also include big methods ( \leq 512 tokens). We noticed, however, that only a minority of the code instances in our dataset contained comments (< 20%) and that many of them are low quality ( e.g., ‘‘// Argh!!! This method is WRONG !’’ ). For this reason, we use an automated procedure to generate higher-quality comments needed in our experiment. Considering the taxonomy of code comments introduced by Zhai et al. ( 69 ) , we use an LLM to generate five comments, documenting what the method does, how it does it, the intended usage of the method, the reason why the method is there, and properties of the method ( e.g., pre-conditions and post-conditions). We do this on the fixed version rather than on the buggy one to simulate the ideal scenario in which developers properly document the methods they develop. To evaluate our hypothesis that code comments play a crucial role in Automated Bug Fixing (ABF), we designed a controlled experiment manipulating the presence of comments both during training and inference. We fine-tuned two types of models—CodeT5+ ( 64 ) and DeepSeek-Coder ( 19 ) —under all combinations of these conditions ( with vs. without comments). For the training phase, comments were either removed from or automatically generated for each instance, while at inference time the models were tested on inputs with or without comments accordingly. When comparing the four training-inference combinations, we found that models benefit from comments in both phases. Comments provided only at inference time already lead to noticeable improvements, while training with comments does not harm performance on instances lacking comments. The greatest improvements occur when comments are present in both phases, increasing accuracy by up to threefold. These results demonstrate that code comments can substantially enhance models performance, underscoring their importance in ABF tasks. To gain deeper insights into the impact of the five categories of comments on model performance, we conducted an interpretability study based on SHAP ( 32 ) , employing the GradientExplainer method. This method assigns an importance score to each input token with respect to every token generated during the inference of a single generation.