CCISolver : End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency
Renyi Zhong, Yintong Huo, Wenwei Gu, Jinxi Kuang, Zhihan Jiang, Guangba Yu, Yichen Li · 2025
Read CCISolver : End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency by Renyi Zhong, Yintong Huo, Wenwei Gu, Jinxi Kuang, Zhihan Jiang, Guangba Yu, Yichen Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20558.txt.
Introduction
Renyi Zhong and co-authors tackle stale notes in computer code, where a comment no longer matches what the code does. Old notes mislead readers and slow fixes and tests. Past tools for the flaw were trained on sets with many wrong labels, which capped their worth. The 2025 paper of the team first audits those old sets and shows the scale of the label problem.
To fix the base, the team built a cleaned set of high-grade cases for training and tests. On top of it they built a tool that both finds a mismatch and rewrites the note in one pass. The tool hit a top score on the finding task and beat the best past tool on the fixing task by a wide step. Human judges also rated its fixes best. It ran about a third faster than the base model, which suits real use.
Catalog record
| CREATOR | Renyi Zhong, Yintong Huo, Wenwei Gu, Jinxi Kuang, Zhihan Jiang, Guangba Yu, Yichen Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20558
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CCISolver : End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency Thanks: Renyi Zhong, Wenwei Gu, Jinxi Kuang, Zhihan Jiang, Guangba Yu, Yichen Li, Michael R. Lyu are with The Chinese University of Hong Kong, Hong Kong. (e-mail:[email protected]; [email protected]; jxkuang22 @cse.cuhk.edu.hk; [email protected]; [email protected]; [email protected]; [email protected]) Yintong Huo and David Lo are with Singapore Management University, Singapore. (e-mail: [email protected]; [email protected]) Renyi Zhong, Yintong Huo, Wenwei Gu, Jinxi Kuang, Zhihan Jiang, Guangba Yu, Yichen Li Abstract Comments within code serve as a crucial foundation for software documentation, facilitating developers to communicate and understand the code effectively. However, code-comment inconsistency (CCI) can negatively affect software development, testing, and maintenance. Recent efforts to mitigate this issue have emerged, but existing studies often suffer from inaccurate datasets and inadequate solutions, weakening their practical effectiveness. In this study, we first conduct a quantitative analysis of existing datasets, revealing a substantial portion of sampled data are mislabeled. To address these data limitations, we introduce CCIBench , a refined dataset comprising high-quality data, to support the training and evaluation of method-level CCI methods. Furthermore, we present an innovative end-to-end LLM-based framework, CCISolver , designed to improve code quality by identifying and rectifying CCIs. Comprehensive evaluations demonstrate CCISolver ’s superior performance. For detection, it establishes a new state-of-the-art with an F1-score of 89.54%. In fixing task, it achieves a remarkable 18.84% relative improvement in GLEU score over the strongest baseline. This superiority is confirmed by human evaluation, where CCISolver ’s fixing success rate of 0.6533 significantly surpasses existing methods. Critically, in a practical end-to-end setting, CCISolver ’s innovative architecture is approximately 36% faster for inference than the baseline model, underscoring its scalability and real-world applicability. I Introduction Code comments serve as fundamental documentation artifacts in software development, providing crucial metadata and explanations that facilitate program comprehension and maintenance. These textual annotations enable developers to communicate essential information, including implementation details, component relationships, and code evolution rationale [ 1 , 2 ] . Research has demonstrated that well-maintained comments can enhance software maintainability and support developers’ understanding of complex codebases [ 3 ] . To ensure that comments align with their related code, it is imperative for comments to be updated alongside code modifications [ 4 ] . However, developers often neglect to revise comments during code changes or refactoring, resulting in code-comment inconsistencies (CCI) where comments misrepresent the code. For instance, Figure 1 illustrates a CCI scenario in spring-data-mongodb, where a method was modified to handle Document objects, yet the comment incorrectly claims it works with DBObject. These inconsistencies can hinder development workflows by misrepresenting code intent and wasting developers’ time interpreting incorrect comments [ 5 ] . Fig. 1 : A CCI example in spring-data-mongodb [ 6 ] . Recognizing the scale of modern systems and the labor-intensive nature of manually resolving CCI, researchers have proposed automated CCI detection techniques. These techniques leverage both traditional machine learning classifiers [ 7 ] and advanced deep learning models [ 5 , 8 , 9 ] . However, there are limitations from both dataset side and proposed solution side. Specifically, existing datasets contain data mislabelling issues originating from the trade-offs made during dataset construction [ 9 ] . Existing deep learning methods [ 5 , 8 ] excel in detecting CCIs but are inherently limited in fixing them, while approaches like C4RLLaMA [ 10 ] that utilize fine-tuned LLMs for both tasks introduce significant computational overhead, particularly during the detection phase. Existing dataset limitations. The current CCI datasets (e.g., JITData [ 8 ] and OcdData [ 5 ] ) contain data mislabeling issues stemming from the trade-offs made during their construction. Researchers often opt for automated labeling over manual labeling due to the latter’s high costs, although it can lead to labeling errors [ 9 , 11 ] . Specifically, inconsistencies are identified by comparing pairs of methods and comments from older and newer code versions, under the assumption that an individual comment update indicates a correction for the method, thus labeling the old comment as inconsistent with the new code. However, this process relies on an overly broad assumption that may not hold true in practice. For example, a comment may be updated to correct typos or add information, which could mistakenly classify it as inconsistent. While previous studies [ 10 , 11 ] have acknowledged the issue of erroneous labeling, no further investigation has been conducted to identify the causes of false positives and their prevalence. Existing solution limitations. A complete CCI solution requires both detecting inconsistencies and fixing them by suggesting appropriate comment revisions [ 10 ] . While existing deep learning methods [ 5 , 8 ] have shown promising result in detection phase, they fall short in fixing CCIs. Recent advancements in large language model (LLM) have demonstrated exceptional capabilities in code understanding and repair [ 12 ] . C4RLLaMA [ 10 ] utilizes a fine-tuned LLM to handle both detection and correction activities. Nonetheless, this method incurs considerable performance overhead because the detection task examines a vast number of cases, most of which do not require further inconsistency fixing. To investigate the potential dataset limitations, we first conduct a preliminary study on the widely-used JITData dataset. Our findings are alarming: a manual inspection of 600 sampled cases revealed that a staggering 45.67% of instances labeled as positive (i.e., inconsistent) are, in fact, incorrectly identified (§ II-B3 ). We pinpoint four primary reasons for these errors: add/delete information , fix typos , change case , and change lexical . These results not only alarm critical data quality issues in existing resources, but also underscore the urgent need for a cleaner dataset. In response to the data quality issues identified, we develop a new dataset, CCIBench , derived from JITData . The construction of CCIBench aims to directly address the identified shortcomings, such as high false positive rates and their underlying causes, to provide a high-quality resource for CCI detection and fixing tasks. This involved a multi-stage process incorporating advanced de-duplication, syntactic cleaning, and semantic-based filtering. Initially, we eliminate duplicates in the data, a common issue that inflates existing datasets [ 9 ] . We then use refined syntactic rules to remove trivial changes (e.g., typo corrections, case adjustments) that do not lead to CCI, directly addressing some of the previously identified false positive sources. To address more complicated cases, particularly those related to changes like information addition or deletion which syntactic rules may miss, we employ three powerful LLMs (i.e., GPT4o [ 13 ] , Claude3.5-Sonnet [ 14 ] , and LLaMA3.1-405b [ 15 ] ) as independent voters to identify true CCIs, resulting in a dataset of 22,360 verified CCI cases. Beyond the critical need for a cleaner dataset, effectively addressing CCIs also requires overcoming the limitations inherent in current solutions. As previously discussed, existing approaches either excel at detection but lack repair capabilities, or utilize computationally intensive LLMs for both tasks, leading to inefficiencies. To bridge this gap, we propose CCISolver , an innovative end-to-end framework designed for both efficient CCI detection and effective automated repair. CCISolver combines the strengths of different methodologies: (1) For detection, the component CCIDetector uses a traditional deep learning approach. It first extracts code comments and associated code changes, then