The Impact of Large Language Models (LLMs) on Code Review Process Journal: Information and Software Technology

Antonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, Emad Shihab · 2025

Read The Impact of Large Language Models (LLMs) on Code Review Process Journal: Information and Software Technology by Antonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, Emad Shihab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11034.txt.

Introduction

Antonio Collante and co-authors measure how GPT help changed code review on GitHub. They built a set of 25,473 pull requests from 9,254 projects and flagged GPT-assisted ones with keyword and pattern checks plus hand checks, reaching ninety-five percent labelling accuracy. Statistical models then compared assisted and plain reviews as a whole and phase by phase.

GPT-assisted reviews closed far faster, with a median of nine hours against twenty-three. Review time fell by thirty-three percent and waiting time before acceptance by eighty-seven percent. In a sample of three hundred assisted reviews, developers used GPT most for code optimization, then bug fixing, then documentation.

Catalog record

CREATORAntonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, Emad Shihab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11034

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Impact of Large Language Models (LLMs) on Code Review Process Journal: Information and Software Technology Antonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, Emad Shihab Abstract Large language models (LLMs) have recently gained prominence in the field of software development, significantly boosting productivity and simplifying teamwork. Although prior studies have examined task-specific applications, the phase-specific effects of LLM assistance on the efficiency of code review processes remain underexplored. [Objectives:] This research investigates the effect of GPT on GitHub pull request (PR) workflows, with a focus on reducing resolution time, optimizing phase-specific performance, and assisting developers. [Methods:] We curated a dataset of 25,473 PRs from 9,254 GitHub projects and identified GPT-assisted PRs using a semi-automated heuristic approach that combines keyword-based detection, regular expression filtering, and manual verification until achieving 95% labeling accuracy. We then applied statistical modeling, including multiple linear regression and Mann–Whitney U test, to evaluate differences between GPT-assisted and non-assisted PRs, both at the overall resolution level and across distinct review phases. [Results:] Our research has revealed that early adoption of GPT can substantially boost the effectiveness of the PR process, leading to considerable time savings at various stages. Our findings suggest that GPT-assisted PRs reduced median resolution time by more than 60% (9 hours compared to 23 hours for non-assisted PRs). We discovered that utilizing GPT can reduce the review time by 33%, and the waiting time before acceptance by 87%. Analyzing a sample dataset of 300 GPT-assisted PRs, we discovered that developers predominantly use GPT for code optimization (60%), bug fixing (26%), and documentation updates (12%). [Conclusion:] Our findings highlight the potential of GPT to accelerate code reviews, improve code quality, and address persistent bottlenecks. This research sheds light on the impact of the GPT model on the code review process, offering actionable insights for software teams seeking to enhance workflows and promote seamless collaboration. 1 Introduction Modern software development is heavily based on collaborative workflows, with Pull Requests (PR) serving as a central mechanism to integrate code changes [ 1 , 2 , 3 ] . A PR goes through multiple phases, including submission, review, change, and merging, each requiring coordination between contributors and reviewers [ 1 ] . Effective PR management is crucial to maintaining software quality, but it remains a bottleneck in many projects due to long review times, inconsistent feedback, and inefficient communication [ 4 ] . Developers often experience delays due to unclear review comments, excessive revisions, and prolonged waiting times before changes are accepted [ 5 ] . Inefficiencies in the software development cycle, such as long review times, can lead to frustration among team members, decreased productivity, and even project delays [ 6 ] . As codebases become increasingly intricate, there is a growing need for sophisticated automation tools to facilitate the code review process and improve collaboration and teamwork [ 7 ] . In parallel, LLMs, such as Generative Pre-trained Transformers (GPTs) [ 8 ] have rapidly evolved, demonstrating strong capabilities in code generation, bug detection, and documentation assistance [ 9 , 10 ] . These models are capable of suggesting enhancements to existing code, resolving unclear logical structures, and automating repetitive tasks, thereby becoming indispensable tools for software engineers. Despite the increasing adoption of LLM-powered tools in development workflows, a systematic investigation into their impact on PR processes is lacking. Previous research on the use of LLM in the PR context [ 9 , 11 , 12 , 13 ] has focused mainly on task-specific applications, such as commit message generation [ 14 ] and documentation support [ 15 ] , while neglecting to explore their phase-specific impact across the entire PR lifecycle. Furthermore, while LLM-generated suggestions can be beneficial, it is necessary to quantify their effectiveness in improving review speed and reducing PR merge time. This study aims to fill this knowledge gap by empirically evaluating the impact of GPT on code review waiting times, with a particular focus on phase-specific effects. It examines their influence on resolution time, their role in different PR phases, and the reasons why contributors and reviewers adopt GPT as assistance. Given the proven effectiveness of GPT on a wide range of software engineering tasks, including code generation, debugging, and documentation, understanding how and to what extent GPT enhances the effectiveness of PRs in each phase is crucial to assess their usefulness in real-world operations. These insights can guide the design of more efficient GPT-supported code review workflows and inform best practices for integrating AI assistance into collaborative software development. Toward this goal, we examine the PRs that software developers actively engaged with GPT to seek recommendations for code review tasks or problem-solving. We refer to these as GPT-assisted PRs throughout the paper. In contrast, GPT non-assisted PRs are those where developers either do not use GPT at all or use it strictly for implementation tasks (i.e., making an API call to a GPT model), without relying on it for code review insights or decision making. Our research investigates how GPT-assisted PRs influence the duration and quality of the code review process. For this purpose, we start by curating a dataset of 25,473 PRs from 9,254 projects on GitHub. Next, we extract six features ( project_age, no_commits, files_added, files_deleted, files_changed, time_to_merge ) to characterize projects, PRs, and code reviews. We then analyze how these factors influence the review and resolution times [ 16 ] . Finally, we also investigate which development tasks involve the use of GPT-assisted PRs throughout the PR phases, aiming to identify where such assistance is most effectively integrated. In summary, our aim is to answer the following three research questions: RQ1: Does GPT-assistance during the PR process reduce the resolution time? We aim to understand the potential of LLMs to expedite the resolution of PRs in the code review process. Specifically, we analyze their impact on the total time required to resolve a PR, from its submission to completion. Our results suggest a correlation between the use of GPT-assisted PRs and a decrease in merge time approximately from 23 to 9 hours . RQ2: How does GPT assistance impact various phases of the PR process? We aim to further explore the effect of GPT in various phases of the PR process, such as submission, evaluation, modification, and waiting time, to determine where its support is most advantageous. Our analysis shows that GPT-assisted PRs complete faster across key phases of the review process. Specifically, they take 1 hour at the Review phase, 3 hours at Waiting for Changes, and 1 hour at Change, compared to 3 hours, 24 hours, and 1 hour, respectively, for non-assisted PRs. These results highlight the potential of GPT to streamline the PR workflow, especially by reducing waiting times. RQ3: Why do contributors and reviewers use GPT during the PR process? We aim to better understand how programmers use GPT to enhance code quality, fix errors, and improve documentation. By analyzing these application patterns, we uncover how developers integrate this technology into their daily workflows, highlighting its most potential benefits. Our findings show that developers devote a significant part of their review time to refining and correcting the source code. Specifically, 60.26% of enhancement tasks occur during the Review phase, 42.31% during

Read the complete text