Consider What Humans Consider: Optimizing Commit Message Leveraging Contexts Considered By Human CCS: Software and its engineering Software maintenance tools

Jiawei Li, David Faragó, Christian Petrov, Iftekhar Ahmed · 2025

Read Consider What Humans Consider: Optimizing Commit Message Leveraging Contexts Considered By Human CCS: Software and its engineering Software maintenance tools by Jiawei Li, David Faragó, Christian Petrov, Iftekhar Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11960.txt.

Introduction

Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed improve human-written commit messages with language models, in two thousand twenty-five. A commit message records why a code change was made, aiding maintenance and team talk. Automated generators miss some context that developers weigh, and that context can be hard to retrieve by machine.

Their method starts from the human message and improves it in rounds. It folds in key context and scores from outside evaluators, using search to guide the edits. In wide tests, the optimized messages were judged more rational, complete, and expressive. They beat both top generators and the original human messages in most head-to-head comparisons.

Catalog record

CREATORJiawei Li, David Faragó, Christian Petrov, Iftekhar Ahmed
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Consider What Humans Consider: Optimizing Commit Message Leveraging Contexts Considered By Human CCS: Software and its engineering Software maintenance tools Jiawei Li, David Faragó, Christian Petrov, Iftekhar Ahmed Abstract . Commit messages are crucial in software development, supporting maintenance tasks and communication among developers. While Large Language Models (LLMs) have advanced Commit Message Generation (CMG) using various software contexts, some contexts developers consider to write high-quality commit messages are often missed by CMG techniques and can’t be easily retrieved or even retrieved at all by automated tools. To address this, we propose Commit Message Optimization (CMO), which enhances human-written messages by leveraging LLMs and search-based optimization. CMO starts with human-written messages and iteratively improves them by integrating key contexts and feedback from external evaluators. Our extensive evaluation shows CMO generates commit messages that are significantly more Rational, Comprehensive, and Expressive while outperforming state-of-the-art CMG methods and human messages 40.3%-78.4% of the time. Moreover, CMO can support existing CMG techniques to further improve message quality and generate high-quality messages when the human-written ones are left blank. 1. Introduction Commit messages are essential in documenting code changes by explaining what was modified and why, as well as providing necessary context for software maintenance. High-quality messages succinctly capture the change details (“What”), the rationale (“Why”), and any additional relevant information, which is vital for assessing the commit’s impact on the codebase ( Tian et al. 2022b ; Li et al. 2024 ) . Well-written messages also help reduce software defect proneness ( Li and Ahmed 2023 ) . However, developers often neglect to write high-quality messages due to time constraints or lack of motivation. This leads to vague or incomplete messages—even in projects under the Apache Software Foundation ( Tian et al. 2022b ; Li and Ahmed 2023 ) . This issue is widespread; for instance, Dyer et al. ( Dyer et al. 2013 ) found that approximately 14% of commit messages in over 23,000 SourceForge projects were entirely blank. To assist developers in writing commit messages and elevate the commit message quality issue, numerous automated Commit Message Generation (CMG) techniques have been proposed by the Software Engineering (SE) research community. These methods, given a set of code changes, aim to generate descriptive commit messages. These techniques can be broadly categorized into Template-based approaches where predefined patterns are used for generating messages ( Linares-Vásquez et al. 2015 ; Shen et al. 2016 ) . Retrieval-based methods which find similar past commit messages and adapt them ( Huang et al. 2020 ; Shi et al. 2022 ) . Translation-based techniques treat commit message generation as a language translation problem, translating code changes into natural language ( Xu et al. 2019 ; Liu et al. 2020 ; Nie et al. 2021 ) . Hybrid approaches combine elements from the above techniques to enhance performance ( Liu et al. 2020 ; Shi et al. 2022 ; He et al. 2023 ) . However, researchers have noted that these methods often produce messages that lack key information, such as the “What” or “Why,” provide insufficient context, follow simple patterns, or are semantically unrelated to the human-written messages and difficult to read ( Jiang et al. 2017 ; Wang et al. 2021 ; Liu et al. 2018 ; Dong et al. 2023 ) . This underscores the need for further improvements in CMG techniques to generate higher-quality messages. Recent advancements in Large Language Models (LLMs) have significantly enhanced various software engineering tasks, including CMG. Li et al. ( Li et al. 2024 ) introduced a state-of-the-art approach leveraging GPT-4 with the ReAct prompting technique ( Yao et al. 2022 ) . Their method, Omniscient Message Generator (OMG), outperformed previous CMG techniques by incorporating a broader range of contextual information. While traditional CMG methods rely solely on code changes ( Liu et al. 2020 ; Shi et al. 2022 ; Dong et al. 2022 ) , OMG integrates additional software contexts, such as pull request titles and method/class summaries. However, since commit messages serve as one of the primary communication channels for developers, facilitating discussions on diverse software maintenance activities that often depend on various software contexts ( Mannan et al. 2020 ; Tian et al. 2022b ; Li and Ahmed 2023 ) , it is plausible that OMG misses certain nuanced contextual information considered by developers to compose high-quality messages. This raises the question: RQ1:What software context information do developers consider when writing high-quality commit messages that existing CMG techniques fail to capture? The variety of contextual information developers consider while writing commit messages makes it difficult for automated tools to capture all the necessary human-considered contexts for generating high-quality commit messages. To address this, integrating human guidance is essential for producing commit messages that reflect these diverse contexts. Human guidance has already proven effective in tasks like code repair and assertion generation ( Böhme et al. 2020 ; Zamprogno et al. 2022 ) , and similar methods could be beneficial in CMG. Eliseeva et al. ( Eliseeva et al. 2023 ) introduced Commit Message Completion (CMC) by using human-typed prefixes to help LLMs generate more accurate commit messages. However, their study did not evaluate whether the generated messages included essential information like the “What”/“Why” information. Furthermore, they only considered the preceding commit to assist in message personalization, overlooking other critical software contexts that can impact commit message quality ( Tian et al. 2022b ; Li and Ahmed 2023 ; Li et al. 2024 ) including class/method containing the changes and related issue report/pull request. Inspired by CMC, we propose a new method called Commit Message Optimization (CMO) to enhance the quality of commit messages by leveraging a search-based optimization approach. Starting with a human-written message, CMO iteratively refines and improves the message to optimize its quality. This method addresses the fact that human-written commit messages can often contain quality issues, such as missing essential information or poor readability ( Tian et al. 2022b ; Li and Ahmed 2023 ) . CMO brings two key advantages; firstly, by starting from a human-written message, the model gains access to the nuanced context a developer would naturally consider for the associated code changes. This includes information that might be impractical for automated systems to retrieve but is readily available in a developer’s mind. Second, using the power of LLMs, CMO can “fix” or optimize the commit message by incorporating widely considered contexts and addressing quality issues that the human developer may miss. The iterative process allows the LLM to enhance the message with more relevant information, improving its clarity and completeness. Since developers often leave commit messages empty due to time and motivation constraints ( Dyer et al. 2013 ) , CMO is designed to address this issue by optimizing an initial message generated through few-shot prompting ( Brown et al. 2020 ) . Additionally, we investigate whether CMO can enhance existing CMG techniques, such as OMG, to improve the quality of generated messages. While CMG techniques have been extensively studied in research, their adoption among software practitioners remains limited ( Eliseeva et al. 2023 ; Dong et al. 2023 ) . Our goal is to bridge this gap by making CMG more practical and effective for real-world software development:

Read the complete text