CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation
Bo Xiong, Linghao Zhang, Chong Wang, Peng Liang · 2025
Read CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation by Bo Xiong, Linghao Zhang, Chong Wang, Peng Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18337.txt.
Introduction
Good commit messages tell a team why code changed, yet many are vague or empty. Bo Xiong and four co-authors tackle the task of writing these messages by machine. Their 2025 paper presents CoRaCMG, a system that helps large language models draft messages from code changes. The key idea is to show the model real past examples from the same project.
CoRaCMG works in three steps: find past change-and-message pairs that match, add them to the prompt, then generate the new message. Tests covered models such as GPT, DeepSeek, and Qwen, scored on four standard measures. One good example lifted scores sharply, with some gains near 80 to 90 percent. Gains faded after three examples. The model seems to learn project terms and writing style from the examples it sees.
Catalog record
| CREATOR | Bo Xiong, Linghao Zhang, Chong Wang, Peng Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-18337
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation Bo Xiong, Linghao Zhang, Chong Wang, Peng Liang Abstract Context : In software development and maintenance, commit messages play a key role in documenting the intent behind code changes, supporting collaboration, review, and maintenance. However, they are often low-quality, vague, or incomplete, limiting their usefulness. Commit Message Generation (CMG) aims to automatically generate descriptive commit messages from code diffs to reduce developers’ effort and improve message quality. Although recent advances in Large Language Models (LLMs) have shown promise in automating CMG, their performance remains limited. Objective : This paper aims to enhance CMG performance by retrieving similar diff-message pairs to guide LLMs to generate commit messages that are more precise and informative. Method : We proposed CoRaCMG, a Co ntextual R etrieval- a ugmented framework for C ommit M essage G eneration, structured in three phases: (1) Retrieve: retrieving the similar diff–message pairs; (2) Augment: combining them with the query diff into a structured prompt; and (3) Generate: generating commit messages corresponding to the query diff via LLMs. CoRaCMG enables LLMs to learn project-specific terminologies and writing styles from the retrieved diff-message pairs, thereby producing high-quality commit messages. Results : We evaluated our method on various LLMs, including closed-source GPT models and open-source DeepSeek models. Experimental results show that CoRaCMG significantly boosts LLM performance across four metrics (BLEU, Rouge-L, METEOR, and CIDEr). Specifically, DeepSeek-R1 achieves relative improvements of 76% in BLEU and 71% in CIDEr when augmented with a single retrieved example pair. After incorporating the single example pair, GPT-4o achieves the highest improvement rate, with BLEU increasing by 89%. Moreover, performance gains plateau after more than three examples are used, indicating diminishing returns. Further analysis shows that the improvements are attributed to the model’s ability to capture the terminologies and writing styles of human-written commit messages from the retrieved example pairs. Conclusions : CoRaCMG provides an effective framework for the CMG task, supported by comprehensive evaluations that demonstrate significant improvements in the quality of generated commit messages. keywords Software Maintenance Commit Message Generation Retrieval Augmented Generation † † credit: Conceptualization, Methodology, Investigation, Data curation, Software, Writing - Original draft preparation † † credit: Conceptualization, Methodology, Investigation, Data curation, Software, Writing - Original draft preparation † † credit: Conceptualization, Methodology, Investigation, Writing - Original draft preparation † † credit: Conceptualization, Methodology, Investigation, Writing - Original draft preparation † † These authors contributed equally to this work. † † corresponding: Corresponding authors. 1 Introduction In software development and maintenance, the Git version control system has been widely used to store and share code. Within Git, commit messages describe and document the code changes made in a commit. These messages record alterations to the source code, help developers understand the changes in the code, and promote efficient collaboration. Therefore, the commit message serves as one of the critical pieces of textual information in the software engineering life-cycle. However, writing commit messages is time-consuming and laborious for developers. Many developers find them tedious and are not motivated to write ( Maalej and Happel 2010 ) . As a result, in real-world development settings, the overall quality of commit messages is often suboptimal. As a recent work reported by Tian et al. 2022 , on average 44% of commit messages did not reach the desired quality, indicating a lack of essential information and the struggle to convey critical details about what the commit did and why. CMG task aims to take the differences between two versions of code as input, typically in the form of a code diff file .diff generated by Git, aka code change, and then generates the corresponding commit message. Initially, rule-based approaches were used in CMG ( Buse and Weimer 2010 ; Vásquez et al. 2015 ; Shen et al. 2016 ) , where predefined rules or templates were utilized for generation. Some retrieval-based approaches leverage information retrieval (IR) techniques to suggest commit messages from similar code diff ( Liu et al. 2018 ; Hoang et al. 2020 ) . With the rapid development of deep learning, many learning-based methods have recently emerged ( Dong et al. 2022 ; Jiang et al. 2017 ; Liu et al. 2019 ; Loyola et al. 2017 ; Xu et al. 2019 ) . These methods treat CMG as a neural machine translation task, in which code diff is the input of a neural network model to generate commit messages as output. In addition, there are some hybrid approaches ( Liu et al. 2022 ; Shi et al. 2022 ; Wang et al. 2021 ) that leverage both IR and deep learning techniques. With the recent advances in LLMs, these models have been increasingly applied to both natural language processing and code-related tasks. Several studies ( Zhang et al. 2024b ; Zhang et al. 2024a ) have begun exploring their potential on the CMG task and have reported promising prospects. Additionally, several publicly available datasets ( Liu et al. 2018 ; Jiang et al. 2017 ; Xu et al. 2019 ; Wang et al. 2021 ) have been widely adopted in recent research. For instance, Jiang et al. 2017 constructed a dataset exclusively comprising commit messages from Java projects, which has been reused in numerous subsequent studies ( Buse and Weimer 2010 ; Dong et al. 2022 ; Zhang et al. 2024a ; Xu et al. 2019 ) . Table 1 presents an overview of several commonly used commit message datasets along with their associated statistical information. These datasets are typically constructed by crawling commit records from open-source projects hosted on GitHub. However, they often suffer from some issues related to data quality and the standardization of their construction processes. A study by Zhang et al. 2024a revealed a noteworthy phenomenon: in a human evaluation experiment, participants were tasked with selecting the optimal commit message from a set of generated messages (including the reference message authored by developers) for a given commit. The experiment utilized a widely adopted commit message dataset ( Xu et al. 2019 ) . Results indicated that, across 366 commit samples, only 13.1% of the samples had the developer-authored reference commit message selected as the optimal choice. This finding underscores the inherent quality issues within these datasets and highlights their potential impact on the standardized evaluation protocols for the automated CMG task. Given the limitations of existing datasets, it is crucial to use a high-quality and comprehensive dataset to fairly evaluate the effectiveness of CoRaCMG. However, directly applying LLMs to CMG still faces several challenges. First, the generated messages may overlook project-specific writing styles. Second, the input context is often limited and lacks a global understanding of the codebase, which can result in missing critical information necessary for accurate generation. Most importantly, existing models have not yet effectively leveraged a key characteristic in software engineering: historical commit messages often contain the terminology of human-written commit messages and the writing styles. Against this backdrop, RAG ( Gao et al. 2023 ; Ram et al. 2023 ; Lewis et al. 2020 ; Wang et al. 2023 ) offers a promising direction for the task of automatic CMG. By retrieving external knowledge to enrich the input with relevant information, RAG enhances the performance of generation